GPU acceleration at scale with OpenPower platforms

We are working to optimise Code_Saturne, an open-source software package developed by EDF for complex CFD simulations to run on next-generation architectures.

Challenge

As computing platforms continue to evolve, the ability for software to run on the next generation of hardware is essential. New architectures feature Graphical Processing Units (GPUs) as standard, and codes need to be optimised to take advantage of their increased performance potential. Code_Saturne is a general purpose Computational Fluid Dynamics (CFD) software package developed by EDF. It can be run on thousands of cores but is difficult to accelerate on GPUs due to unstructured data layouts, memory-bound algorithms and a large code base. To be able to solve more complex CFD problems, researchers must find a way to move data between Central Processing Units (CPUs) and GPUs more efficiently, enabling the software to take advantage of the power offered by next generation platforms.

Approach

Simulating a 3D flow in a cavity as a test, the team combined compute tasks scattered in the code using the programming language CUDA, allowing tasks to be queued simultaneously. They also explored ways to keep data on the GPU for longer, enhancing performance by overcoming bottlenecks from moving data between CPUs and GPUs. To improve occupancy of GPUs and take advantage of their capabilities, the team combined the NVIDIA Multiple Process Service (MPS) and dual buffering techniques. Getting a single GPU to work with multiple CPUs at any given time enables larger CFD problems to be solved.​​

Benefits

Optimising Code_Saturne for GPU acceleration meant that CFD simulations ran up to three times faster compared to those run solely on CPUs. This paves the way for organisations to run more complex and high fidelity simulations, providing greater confidence in the solutions offered. Our work showed that GPU-accelerated Code_Saturne has the ability to scale to IBM Summit, the world’s most powerful supercomputer. This demonstrates the art of what is possible in CFD simulation, allowing larger and more complex problems to be solved by applying Open Source Code_Saturne software.​​​

“These developments could encourage a broader use of GPUs on our clusters, and adaptations for machines such as IBM’s Summit are an excellent showcase of the applications of this technology. We hope to do our best to ensure these developments remain available in Code_Saturne for the CFD community going forward. This has shown that it is possible to efficiently port Code_Saturne to nodes with GPUs without conflicting the code’s portability and ability to be maintained, while also improving performance even for memory-bound codes.​​​​​​​​”

Stéphane Tanguy, EDF​​

Join Newsletter

Provide your details to receive regular updates from the STFC Hartree Centre.