Multi-Region training with Amazon SageMaker HyperPod and Qumulo

With Amazon SageMaker HyperPod and Qumulo, you can place training compute in one AWS Region and keep your dataset in another. Training large AI models requires massive GPU capacity, but your ideal compute resources and your training data don’t always reside in the same AWS Region. Accessing data across Regions adds network latency and transfer costs. Teams face a choice: either replicate petabytes of data across Regions, or absorb cross-Region latency on every read and accept slower training. This pairing can help tackle that trade-off, letting teams keep frontier models current without moving data or sacrificing throughput.

Pairing Amazon SageMaker HyperPod with CNQ delivers two measurable outcomes:

• Single-Region training performance: CNQ’s cloud-native architecture allows it to scale its performance independently of its data storage size. This results in full 99 percent GPU utilization and p5.48xlarge network saturation with sub-3 ms data operations.

Solution overview

Amazon SageMaker HyperPod provides the managed training infrastructure including resilient clusters with automatic health checks, node replacement, and checkpoint recovery. The solution pairs SageMaker HyperPod with Qumulo’s storage layer. We validated the solution with the following components:

• Single source of truth: Store your training dataset in one AWS Region using Cloud Native Qumulo (CNQ).
• Cross-Region access: Each HyperPod cluster mounts its local CNQ instance over Network File System (NFS). The CNQ spoke cluster retrieves data from the hub cluster over Virtual Private Cloud (VPC) peering, making the full dataset available without cross-Region access from the compute nodes.
• Intelligent data placement: Qumulo’s Cloud Data Fabric (CDF) uses predictive caching through NeuralCache. It learns your data loader access patterns and serves data from local Non-Volatile Memory Express (NVMe), making cross-Region latency transparent after an initial warmup period.

Figure 1 shows the validated architecture. In the spoke Region, machine learning developers submit jobs through an Amazon Elastic Kubernetes Service (Amazon EKS) orchestrator, which runs them on the HyperPod cluster compute nodes. Those nodes mount a local CNQ spoke over NFS. The CNQ hub in the second Region holds the single copy of the training data, and CDF projects that dataset to the spoke over VPC peering, so the compute nodes read from the same local mount path in both Regions.

Figure 1: SageMaker HyperPod with multi-Region data on Qumulo

About Qumulo Cloud Data Fabric

CDF makes a single dataset on a hub Cloud Native Qumulo (CNQ) instantly visible and accessible to multiple spoke CNQ instances. This holds even on network links with more than 100 ms of latency. The primary component is Qumulo’s NeuralCache, which uses an AI model to predict the next 4 KB blocks a job will need and pre-caches them at the spoke. Combined with CNQ’s elastic performance, the dataset performs like local storage for the spoke clusters in this validated configuration, without the costs of full data replication.

We ran the same training job (1.02 billion-parameter LLaMA v3, two ml.p5.48xlarge instances per Amazon SageMaker HyperPod cluster, 16 H100 GPUs total) independently on each cluster. The following table compares the performance:

Configuration	Training speed	Time to 999 batches	NeuralCache state
Hub (us-east-2 to us-east-2)	116–117 samples/sec	18.5 minutes	N/A (local)
Spoke, Warm Cache (us-east-2 to us-west-2)	115–116 samples/sec	18.5 minutes	Converged
Spoke, Cold Start (us-east-2 to us-west-2)	95–115, then 116 samples/sec	~19–20 minutes*	Warming then Converged

** Worst-case scenario with no prior runs. Batches 0–150 run 15–20 percent slower during NeuralCache warmup, then converge to hub-level performance within the first epoch.*

Understanding the architecture

The following sections introduce the validated architecture with Qumulo configured in the same Region as the HyperPod cluster as a hub and also as a spoke.

How Cloud Data Fabric and NeuralCache work

Cloud Data Fabric (CDF) keeps a single authoritative copy of your dataset on the hub cluster and projects it to spoke clusters as a unified Portable Operating System Interface (POSIX) namespace. When a spoke is created, CDF replicates the filesystem metadata to it, so the full namespace is browsable within seconds, before file data has moved. When the spoke mounts the filesystem, CDF services each read from the nearest valid source: local NVMe cache on a hit, the hub on a miss. CDF also uses advanced congestion control, which paces to measured bottleneck bandwidth and round-trip propagation time rather than backing off on packet loss. This is what keeps a long-haul link full where loss-based algorithms collapse. CDF sustains near-line-rate throughput on paths with round-trip times (RTTs) as high as 900 ms.

NeuralCache is the predictive caching layer inside CDF. It observes the sequential 4 KB block reads issued by the training data loader and learns the access pattern. It then places data predictively before the job issues the request. Within the first 100–150 batches, the model learns to predict which blocks will be requested next and pre-fetches them from the hub into local NVMe storage on the spoke. Once warmed, 94–96 percent of reads are served from local cache at sub-5 ms latency. This results in lower latency operations for applications.

Hub configuration

Data and compute share the same Region. Qumulo delivers 1.0–1.3 GBps sustained read throughput (saturating the 10 Gbps network link) at 2–3 ms max read latency, keeping GPUs fully utilized throughout training. Co-locating data and compute matters because every data loader read travels only within the Region, so there’s no wide-area network hop on the critical path. CNQ sustains these numbers because its cloud-native architecture scales throughput with the size of the cluster rather than the size of the dataset. This is why a 4-node cluster can saturate the network link feeding the GPUs. Figure 2 shows this single-Region layout, with the HyperPod cluster, the Amazon EKS orchestrator, and the CNQ hub that stores the dataset all in one Region.

Figure 2: SageMaker HyperPod with single-Region data on Qumulo

Spoke configuration (cold start)

Your Amazon SageMaker HyperPod cluster runs in a different Region from where the data originates. It mounts the local Qumulo spoke, which handles fetching data from the hub. NeuralCache observes your data loader access patterns in real time and begins caching data locally. During the initial 100–150 batches, cross-Region latency is visible and training runs at 95–105 samples per second. Performance then converges to hub-equivalent levels (115–117 samples per second) for the remainder of the run. Figure 3 shows this cross-Region layout. The HyperPod cluster and a CNQ spoke run in one Region, and the CNQ hub that holds the dataset runs in a second Region, connected by CDF over VPC peering.

Figure 3: SageMaker HyperPod with multi-Region data on Qumulo (cold start, worst case)

Spoke configuration (warm cache)

Subsequent runs benefit from a fully warmed NeuralCache. With a 94–96 percent cache hit rate, data is served from local NVMe at sub-5 ms latency, delivering immediate hub-equivalent performance with GPU utilizations above 99 percent.

Walkthrough

This walkthrough provides a step-by-step process for configuring and running these same tests.

Prerequisites

To set up a representative environment, you need the following prerequisites:

• An AWS account with Amazon SageMaker HyperPod enabled.
• Deploy Qumulo Cloud Native in your primary Region (us-east-2, Ohio).
• Sufficient quota for ml.p5.48xlarge instances (two per cluster).
• Configure VPC peering between hub and spoke Regions.
• Basic familiarity with PyTorch and distributed training concepts.

Step 1: Deploy Qumulo Cloud Native in your hub Region

Deploy Qumulo Cloud Native in us-east-2 from AWS Marketplace. In the Amazon Simple Storage Service (Amazon S3) bucket that hosts your deployment files, find the AWS CloudFormation template at templates/cnq-standard.template.yaml and copy its object URL. Create the stack from that URL, then choose r5.8xlarge instances and a 4-node cluster in your existing VPC.

Step 2: Configure VPC peering between Regions

Create a VPC peering connection between your hub (us-east-2) and spoke (us-west-2) VPCs. Then accept the connection in the peer Region and update the route tables in both Regions so that each VPC’s CIDR range routes to the peering connection. This gives Qumulo instances in the spoke Region a path to the Qumulo instances in the hub Region. Now, Amazon SageMaker HyperPod instances in the spoke Region (us-west-2) have access to data in the hub Region (us-east-2) through their local NFS mount. This is because both Qumulo clusters are connected with a private, low-latency path.

VPC peering traffic between AWS Regions is encrypted in transit. Configure security groups on both the Qumulo and HyperPod instances to allow only the required NFS ports (TCP 2049) between the two VPCs.

Note: Replace placeholder values (shown in angle brackets) with your own resource identifiers.

aws ec2 create-vpc-peering-connection \
    --vpc-id <hub-vpc-id> \
    --peer-vpc-id <spoke-vpc-id> \
    --peer-region us-west-2 \
    --region us-east-2

Accept the peering connection from the spoke Region. Then update the route tables in both Regions so that each VPC’s CIDR block routes through the peering connection:

# Accept the peering connection in the spoke Region (us-west-2)
aws ec2 accept-vpc-peering-connection \
    --vpc-peering-connection-id <peering-connection-id> \
    --region us-west-2

# Add a route in the hub route table (us-east-2) pointing to the spoke CIDR
aws ec2 create-route \
    --route-table-id <hub-route-table-id> \
    --destination-cidr-block <spoke-vpc-cidr> \
    --vpc-peering-connection-id <peering-connection-id> \
    --region us-east-2

# Add the reciprocal route in the spoke route table (us-west-2)
aws ec2 create-route \
    --route-table-id <spoke-route-table-id> \
    --destination-cidr-block <hub-vpc-cidr> \
    --vpc-peering-connection-id <peering-connection-id> \
    --region us-west-2

# Verify the peering connection is active
aws ec2 describe-vpc-peering-connections \
    --vpc-peering-connection-ids <peering-connection-id> \
    --query "VpcPeeringConnections[0].Status.Code" \
    --region us-east-2
# Expected output: "active"

Step 3: Upload your training dataset to Qumulo

Mount Qumulo over NFS and sync your pre-tokenized dataset. We used the C4 dataset, tokenized for LLaMA (48.4 million sequences packed to 4,096-token windows). Qumulo presents a unified POSIX namespace, so hub and spoke instances use the same mount path.

sudo mount -t nfs4 -o rw,hard,nointr,proto=tcp <qumulo-ip>:/ /mnt/qumulo
aws s3 sync s3://<your-bucket-name>/c4-tokenized/ /mnt/qumulo/data/c4/

Step 4: Create your HyperPod clusters

Create Amazon SageMaker HyperPod clusters in both Regions with ml.p5.48xlarge instances. HyperPod manages the underlying infrastructure (health monitoring, automatic node replacement, and checkpoint-based recovery), so your training run survives hardware failures without manual intervention. Configure a lifecycle script that mounts Qumulo at startup. The spoke cluster uses the same lifecycle script and mounts the local Qumulo spoke instance over NFS. The Qumulo spoke retrieves data from the hub transparently over VPC peering.

aws sagemaker create-cluster \
    --cli-input-json file://hub-cluster-config.json \
    --region us-east-2

aws sagemaker create-cluster \
    --cli-input-json file://spoke-cluster-config.json \
    --region us-west-2

Each cluster is defined by a CreateCluster request file. The following hub-cluster-config.json creates a two-node ml.p5.48xlarge instance group attached to your Amazon EKS control plane, with automatic node recovery turned on and a lifecycle script that mounts Qumulo at startup:

{
    "ClusterName": "hyperpod-hub-us-east-2",
    "InstanceGroups": [
        {
            "InstanceGroupName": "training",
            "InstanceType": "ml.p5.48xlarge",
            "InstanceCount": 2,
            "LifeCycleConfig": {
                "SourceS3Uri": "s3://sagemaker-<your-bucket-name>/lifecycle-scripts/",
                "OnCreate": "on_create.sh"
            },
            "ExecutionRole": "arn:aws:iam::<account-id>:role/<hyperpod-execution-role>",
            "ThreadsPerCore": 1,
            "OnStartDeepHealthChecks": ["InstanceStress", "InstanceConnectivity"]
        }
    ],
    "VpcConfig": {
        "SecurityGroupIds": ["<hub-security-group-id>"],
        "Subnets": ["<hub-private-subnet-id>"]
    },
    "Orchestrator": {
        "Eks": {
            "ClusterArn": "arn:aws:eks:us-east-2:<account-id>:cluster/<hub-eks-cluster-name>"
        }
    },
    "NodeRecovery": "Automatic"
}

Create spoke-cluster-config.json from the same structure, changing ClusterName, the Region in the Amazon EKS cluster ARN, and the VpcConfig values to the spoke Region resources. Each HyperPod cluster must sit in the same VPC as its Amazon EKS control plane, and the subnets must be private.

Step 5: Configure the Container Storage Interface (CSI) driver to mount Qumulo

Install the Qumulo CSI driver using your preferred installation method. This example uses Helm:

helm repo add qumulo https://qumulo.github.io/qumulo-csi-driver
helm repo update
helm install qumulo-csi qumulo/qumulo-csi-driver --namespace kube-system

Verify the CSI driver pods are running before proceeding to mount:

kubectl get pods -n kube-system -l app=qumulo-csi

# All pods should report STATUS=Running. If any pod shows Pending or CrashLoopBackOff, run to inspect the events:
kubectl describe pod <pod-name> -n kube-system

Create a Static Persistent Volume like this YAML: The accessModes value of ReadWriteMany lets every training node mount the same volume at once, which is what allows the hub and spoke clusters to share one dataset. The server attribute points at the local Qumulo instance in each Region. The same manifest works in both hub and spoke clusters by changing only that value to the local Qumulo address.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-qumulo-static
spec:
  capacity:
    storage: 50Ti
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  mountOptions:
    - vers=3
    - proto=tcp
    - nolock
  csi:
    driver: nfs.csi.k8s.io
    # Unique ID for this directory/volume; use a stable path-based handle
    volumeHandle: qumulo-nfs-root
    volumeAttributes:
      server: <qumulo-spoke-address>
      share: "/"
      #storeExportPath: "/csi"
      #storeRealPath: "/data"
    # If your driver requires secrets for node stage/publish, uncomment:
    nodeStageSecretRef:
      name: cluster1-login
      namespace: kube-system

Finally, create the Persistent Volume Claim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-qumulo-static
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 50Ti
  storageClassName: ""
  volumeName: pv-qumulo-static

Step 6: Run training on your Amazon SageMaker HyperPod clusters

Deploy a Test Pod using kubectl with the following YAML:

apiVersion: v1
kind: Pod
metadata:
  name: qumulo-test
spec:
  containers:
    - name: app
      image: busybox
      command: ["sleep", "3600"]
      volumeMounts:
        - mountPath: /csi
          name: storage
  volumes:
    - name: storage
      persistentVolumeClaim:
        claimName: pvc-qumulo-static

Launch distributed training using PyTorch torchrun, pointing dataset and checkpoint paths to the Qumulo mount. No code changes are needed between hub and spoke runs. The following job.yaml defines the run as a Kubeflow PyTorchJob with one worker per ml.p5.48xlarge instance. It mounts the Qumulo PersistentVolumeClaim at /csi and passes the dataset and checkpoint paths on that mount to the training script:

apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: llama-fsdp
spec:
  pytorchReplicaSpecs:
    Worker:
      replicas: 2
      restartPolicy: OnFailure
      template:
        spec:
          containers:
            - name: pytorch
              image: <account-id>.dkr.ecr.us-east-2.amazonaws.com/<training-image>:latest
              resources:
                limits:
                  nvidia.com/gpu: 8
                  vpc.amazonaws.com/efa: 32
              command:
                - torchrun
                - --nproc_per_node=8
                - --nnodes=2
                - /workspace/train_fsdp.py
                - --dataset-path=/csi/data/c4
                - --checkpoint-path=/csi/checkpoints/llama-fsdp
                - --batch-size=24
                - --seq-length=4096
                - --num-workers=64
                - --prefetch-factor=4
              volumeMounts:
                - mountPath: /csi
                  name: storage
          volumes:
            - name: storage
              persistentVolumeClaim:
                claimName: pvc-qumulo-static

Apply the manifest to the cluster:

kubectl apply -f job.yaml

Configure 64 data loader workers with prefetch_factor=4, keeping 256 batches in flight. This masks latency during cache warmup while Fully Sharded Data Parallel (FSDP) overlaps gradient synchronization with I/O.

Performance results

The performance results for relevant tests are shown in the following sections, illustrating I/O operations per second (IOPS), throughput, and latency per test run. All tests use PyTorch 2.1 FSDP, batch size 24 per GPU, and 4,096-token sequence length.

Hub configuration

The hub delivers consistent performance throughout training: 1.0–1.3 GBps sustained read throughput, 2–3 ms read latency, and 98–100 percent GPU utilization. These numbers are the baseline that the spoke configurations are measured against, because the hub has no wide-area network in the data path. The Qumulo dashboard (Figure 4) displays a stable IOPS profile across a full training run, with the periodic spikes corresponding to checkpoint writes rather than read stalls. The dashboard stacks three time series over the same window: IOPS, throughput, and latency. Throughput holds a flat band just above 1.0 GBps for the length of the run, and the latency trace stays flat alongside it.

Figure 4: Qumulo dashboard showing hub training (us-east-2) with sustained IOPS and throughput

Spoke configuration with cold start (worst case)

NeuralCache begins learning from batch zero. As it observes the data loader, it identifies the sequential access pattern that FSDP training produces and starts prefetching the next blocks before the job requests them, which minimizes data operation latency. The cache hit rate climbs steadily. It stabilizes after the prefetcher is reliably staying ahead of the data loader. Convergence lands around batch 100–150 rather than at a fixed point, because it depends on how quickly the access pattern becomes predictable. These two phases illustrate this transition.

• Phase 1 (batches 0–100): cross-Region latency dominates at over 60 ms, GPU utilization dips to 80–90 percent, and throughput runs at 500–800 MBps.
• Phase 2 (batches 100–150): cache hit rate stabilizes, latency drops to 5–10 ms and training speed climbs to approximately 115 samples per second and 98–100 percent GPU utilization.

Figure 5 shows the same transition on the Qumulo dashboard. IOPS and throughput ramp upward through the warmup window while read latency falls, and all three traces flatten once NeuralCache converges.

Figure 5: Qumulo dashboard showing spoke cold start (us-west-2 to us-east-2) with NeuralCache convergence

At production scale, this one-time penalty diminishes rapidly for geo-distributed training:

Training Scale	Total Batches	Wall Clock Time	Cold Start Impact
Development run	10,000	~3 hours	0.81%
Production training*	100,000	~31 hours	0.08%
Large-scale pretraining*	1,000,000	~13 days	0.008%

** “Production training” and “Large-scale pre-training” numbers are extrapolated from observed cold-start duration. Actual results will vary by dataset size, access pattern, and network conditions.*

Clean up resources

This walkthrough provisions infrastructure in two Regions that continues to incur charges until you remove it. When you finish, tear down the resources in roughly the reverse order you created them so that dependencies are released cleanly.

• Delete the PersistentVolumeClaim and PersistentVolume, then uninstall the Qumulo CSI driver (helm uninstall qumulo-csi -n kube-system).
• Delete both Amazon SageMaker HyperPod clusters (aws sagemaker delete-cluster) in us-east-2 and us-west-2. For the full procedure, see Deleting a SageMaker HyperPod cluster.
• Delete the VPC peering connection and remove the routes you added to the route tables in both Regions. For the full procedure, see Delete a VPC peering connection.
• Delete the Qumulo Cloud Native deployment by removing its AWS CloudFormation stack, which also releases the underlying instances and storage. Before deleting the stack, verify any data you want to keep has been copied off the cluster. The CNQ storage volumes are deleted with the stack and are not recoverable. To delete, open the AWS CloudFormation console, select the stack you created for Qumulo in us-east-2, and choose Delete, as described in Delete a stack from the CloudFormation console. Alternatively, run aws cloudformation delete-stack --stack-name <qumulo-stack-name> --region us-east-2.

Conclusion

By pairing CNQ and CDF with Amazon SageMaker HyperPod, you can run geo-distributed training from a single dataset without data replication. Our validation confirmed that a single copy of your data, with no replication overhead, delivers near-local latency and hub-equivalent throughput after a brief warmup. To avoid the cold start entirely, pre-warm the spoke before training begins by issuing a pre-warm request through the Qumulo API on the dataset inside your training job orchestration script with Ray, Amazon EKS, Slurm, or other tooling.

To extend this architecture, consider multi-spoke deployments accessing a single dataset from three or more Regions, hybrid configurations using AWS Direct Connect for on-premises data, or lifecycle policies to archive checkpoints to an Amazon S3 Glacier storage class.

To get started, see Orchestrating SageMaker HyperPod clusters with Amazon EKS in the Amazon SageMaker AI Developer Guide. For the AWS CloudFormation templates, Terraform modules, and lifecycle scripts that provision clusters like the ones in this post, see the SageMaker HyperPod EKS reference architecture in the awsome-distributed-ai GitHub repository.

For more reading on Cloud Native Qumulo, read Building a Cloud Native File System with Cloud Native Qumulo and Amazon S3.

For more on the components used here, see Introducing Amazon EKS support in Amazon SageMaker HyperPod and Accelerate pre-training of Mistral’s Mathstral model with highly resilient clusters on Amazon SageMaker HyperPod on the AWS Artificial Intelligence Blog. Leave a comment to tell us how you use multi-Region training.

About the authors

Bryan Berezdivin

Bryan is Director of AI Product and Strategy at Qumulo. Prior to Qumulo, he spent over six years at AWS partnering with 100s of AI/ML customers to modernize their data architectures and AI/MLOps, including autonomous systems teams building data frameworks at scale. Bryan also led the launch of AWS multimodal intelligence services purpose-built for AI/ML.

Marcos Perez Seoane

Marcos is a Principal Solutions Architect at Qumulo with over 28 years of experience in enterprise storage. He has worked across leading vendors specializing in SAN, NAS, and object storage, spent three years at AWS as an Amazon FSx for Lustre specialist, and now serves as an SME for Life Sciences workloads, helping organizations power high-performance, data-intensive environments from the edge to the cloud.

Florian Gauter

Florian is a Machine Learning and Generative AI specialist with deep expertise in large-scale distributed training and building AI systems at scale. With a background in data science, he helps organizations architect, optimize, and scale complex AI/ML workloads, from distributed model training to production-ready generative AI.

Mark Vinciguerra

Mark specializes in solving the hard problems of large-scale GPU computing, including distributed training and inference across tens to thousands of accelerators. His expertise spans cluster orchestration with Kubernetes and Slurm, high-performance networking with EFA and NCCL, and storage architectures built to support demanding AI and Generative AI workloads at scale.