FMM electrostatics for biomolecular simulations in GROMACS 

FMM electrostatics for biomolecular simulations in GROMACS
 

We target a flexible, portable and scalable solver for potentials and forces, which is a prerequisite for exascale applications in particle-based simulations with long-range interactions in general. As a particularly challenging example that will prove and demonstrate the capability of our concepts, we use the popular molecular dynamics (MD) simulation software GROMACS. MD simulation has become a crucial tool to the scientific community, especially as it probes time- and length scales difficult or impossible to probe experimentally. Moreover, it is a prototypic example of a general class of complex multiparticle systems with long-range interactions.

MD simulations elucidate detailed, time-resolved behaviour of biology’s nanomachines. From a computational point of view, they are extremely challenging for two main reasons. First, to properly describe the functional motions of biomolecules, the long-range effects of the electrostatic interactions must be explicitly accounted for. Therefore, techniques like the particle-mesh Ewald method were adopted, which, however, severely limits the scaling to a large number of cores due to global communication requirements. The second challenge is to realistically describe the time-dependent location of (partial) charges, as e.g. the protonation states of the molecules depend on their time-dependent electrostatic environment. Here we address both tighly interlinked challenges by the development, implementation, and optimization of a unified algorithm for long-range interactions that will account for realistic, dynamic protonation states and at the same time overcome current scaling limitations.

Download and test our GPU-FMM for GROMACS

If you want to give our GPU-FMM a test drive, please download the tar archive below, unpack with tar -xvzf, and install just like a usual GROMACS 2019.

Our CUDA FMM can be used as a PME replacement by choosing coulombtype = FMM in the .mdp input parameter list. The tree depth d and the multipole order p are set with fmm-override-tree-depth and fmm-override-multipole-order input parameters, respectively. On request (provide your ssh key), the code can be checked out from our git repository git@fmsolvr.fz-juelich.de:gromacs.

GROMACS with GPU-FMM including benchmark systems

Using a multipole order of 8 will result in energies, forces, and an energy drift about as accurate as from a typical PME setting (Coulomb cutoff 1 nm, grid spacing 0.12 nm, order 4, and Ewald rtol parameter 1e-4). You can override the multipoleorder from the .tpr file on the command line with the MULTIPOLEORDER environment variable: MULTIPOLEORDER=8 gmx mdrun -s in.tpr

With sparse systems as the aerosol system, you should set the following environment variable for optimum performance:

export FMM_SPARSE=1

While GROMACS no longer supports open boundaries since the introduction of the Verlet-cutoff scheme, the environment variable OPENBOUNDARY=1 can be set to calculate FMM-based Coulomb interactions with open boundaries. This can make sense for droplet systems, for example, where there is only vacuum at the box edges anyway.

Running FMM in standalone mode

You can also compile and run the GPU-FMM without GROMACS integration. The relevant code is in the ./src/gromacs/fmm/fmsolvr-gpu subdirectory of the above tar archive after unpacking. Compile it with a script like this:

; in bash
export CC=$( which gcc )
export CXX=$( which g++ )
cmake -H../git-gromacs-gmxbenchmarking/src/gromacs/fmm/fmsolvr-gpu -B. -DFMM_STANDALONE=1 -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-10.0
make

The python script runfmm.py can be used to benchmark the standalone version of the GPU-FMM.

Publications

Kohnke, B.; Kutzner, C.; Grubmüller, H.: A GPU-accelerated fast multipole method for GROMACS: Performance and accuracy. Journal of Chemical Theory and Computation 16 (11), pp. 6938 - 6949 (2020)
Kohnke, B.; Kutzner, C.; Beckmann, A.; Lube, G.; Kabadshow, I.; Dachsel, H.; Grubmüller, H.: A CUDA fast multipole method with highly efficient M2L farfield evaluationfield evaluation. The International Journal of High Performance Computing Applications 35 (1), pp. 97 - 117 (2021)
Kohnke, B.; Ullmann, R. T.; Beckman, A.; Kabadshow, I.; Haensel, D.; Morgenstern, L.; Dobrev, P.; Groenhof, G.; Kutzner, C.; Hess, B. et al.; Dachsel, H.; Grubmüller, H.: GROMEX: A scalable and versatile fast multipole method for biomolecular simulation. In: Software for Exascale Computing - SPPEXA 2016-2019, pp. 517 - 543 (Eds. Bungartz, H.-J.; Reiz, S.; Uekermann, B.; Neumann, P.; Nagel, W. E.). Springer, Cham (2020)
Donnini, S.; Ullmann, T.; Groenhof, G.; Grubmüller, H.: Charge-neutral constant pH molecular dynamics simulations using a parsimonious proton buffer. Journal of Chemical Theory and Computation 12 (3), pp. 1040 - 1051 (2016)
Garcia, AG; Beckmann, A; Kabadshow, I
Accelerating an FMM-Based Coulomb Solver with GPUs
Lecture Notes in Computational Science and Engineering 113 (2016) "Software for Exascale Computing - SPPEXA 2013-2015", Eds. HJ Bungartz, P. Neumann, WE Nagel, Springer, pp. 485-504
Páll, S.; Abraham, M. J.; Kutzner, C.; Hess, B.; Lindahl, E.: Tackling exascale software challenges in molecular dynamics simulations with GROMACS. In: Solving Software Challenges for Exascale: International Conference on Exascale Applications and Software, EASC 2014, Stockholm, Sweden, April 2-3, 2014, Revised Selected Papers, pp. 3 - 27 (Eds. Markidis, S.; Laure, E.). Springer, Cham (2015)
A. Beckmann, I. Kabadshow
Portable Node-Level Performance Optimization for the Fast Multipole Method
Lecture Notes in Computational Science and Engineering 105, 29-46 (2015)
A. Arnold, F. Fahrenberger, C. Holm, O. Lenz, M. Bolten, H. Dachsel, R. Halver, I. Kabadshow, F. Gähler, F. Heber, J. Iseringhausen, M. Hofmann, M. Pippig, D. Potts, and G. Sutmann
Comparison of scalable fast methods for long-range interactions
Phys. Rev. E 88 063308-1-22 (2013)
Donnini, S.; Tegeler, F.; Groenhof, G.; Grubmüller, H.: Constant pH molecular dynamics in explicit solvent with lambda-dynamics. Journal of Chemical Theory and Computation 7 (6), pp. 1962 - 1978 (2011)
Go to Editor View