A Practical Guide to GPU-Initiated Communication for Molecular Dynamics at Scale

Molecular dynamics (MD) simulations are among the most demanding workloads in computational science. Using them, researchers can observe atomic behavior in extraordinary detail, from protein folding to drug and materials discovery.

But they come at a steep cost. Simulations model hundreds of thousands to tens of millions of atoms, advancing femtoseconds per step across billions of steps. Because problem size is typically fixed, parallelization must spread work across all available resources simultaneously, a regime known as strong scaling.

GROMACS is one of the world’s most widely used MD packages. Modern hardware and heterogeneous CPU-GPU parallelization has pushed performance into the sub-millisecond regime, reaching 100–200 microseconds per time-step across multiple GPUs. Achieving strong scaling at this level demands extremely low kernel latency, but GPU-to-GPU communication remains a fundamental bottleneck as simulations scale across more GPUs.

Most large-scale HPC applications, including GROMACS, use the Message Passing Interface (MPI) for inter-process communication. However, MPI was designed for CPU-centric execution.

When GROMACS runs on GPUs, the communication workflow forces the GPU to pause while the CPU orchestrates data transfers before signaling the GPU to resume. In GROMACS’s halo exchange—the algorithm that shares boundary atom data between neighboring GPU domains—this handoff repeats across all three spatial dimensions, consuming more than 50% of total CPU wall time at peak iteration rates and capping scalability.

In this post, we examine strategies to eliminate this bottleneck by replacing CPU-orchestrated MPI with GPU-native communication using NVSHMEM.

Accelerating GROMACS using Device-initiated Remote Memory Access

We address both with fine-grained per-pulse dependencies in the Replacing host-side communication and Enabling parallelism across pulses sections.