GPU-SPHysics:

a GPU-based Smoothed Particle Hydrodynamics model for free surface flows



Introduction

GPU-SPHysics is an implementation of Smoothed Particle Hydrodynamics for free surface flows (Monaghan, 1994, 2005). In this application, the fluid can be water or a more viscous substance like lava.

Origin

The Istituto Nazionale di Geofisica e Vulcanologia (INGV) is involved in the LAVA project - "Realization of the lava flow invasion hazard map at Mt Etna and methods for its dynamic update" - co-funded by National Civil Protection (DPC), and coordinated by Ciro Del Negro (INGV) and Stefano Gresta (Università di Catania).

LAVA will provide practical forecasts of the future path of lava flows to enable quantitative hazard assessments and operational guidelines for, potentially, mitigatory actions to be undertaken. We plan to achieve these forecasts by use of numerical computer simulation of the flow paths over the surface of the volcano. These simulations will be constrained by knowledge from former eruptions and from near real-time field and remote observations of the state of lava flow advance.

Within the numerical computer simulation task, INGV- Catania Section, Sicily, is developing a SPH model for lava flow simulation. Due to the high computational requirements of this kind of model it was decided to implement it on GPU.

The first step was to develop a GPU version of the open source SPH free surface flows code SPHysics , which is written in Fortran and exists in serial and parallel form. GPU-SPHYsics uses nearly the same equations and is able to do the same problems, but it uses Nvidia graphics cards to do the heavy computations associated with SPH and thus there are tremendous speed-ups associated with the model. Additionally GPU-SPHysics can handle thermal aspects of the flow.

The project was initiated by Alexis Hérault, coordinator of a LAVA project team (Hérault, 2008; Hérault, Bilotta, and Dalrymple, 2009) from INGV - Catania Section. Additional work has been done by the LAVA project Research Unit of Dipartimento di Matematica e Informatica (Università di Catania) and by Robert A. Dalrymple (Johns Hopkins University).

Some GPU-SPHysics References

Hérault, A., G. Bilotta, and R.A. Dalrymple, SPH on GPU with CUDA, Journal of Hydraulic Research and Engineering, IAHR, sub judice.

From the IV SPHERIC Workshop in Nantes, France:

Hérault et al., Modeling Water Waves in the Surf Zone with GPU-SPHysics, 2009
Dalrymple and Herault, Levee Breaching with GPU-SPHysics Code, 2009

GPU-SPHysics

All the SPHysics examples can be done with GPU-SPHysics, with a faster execution thanks to the usage of the GPU. GPU-SPHysics has been used with the Nvidia GT 8600 (32 streaming processors), GT 8800 (112 streaming processors), the GTX 280 (240 streaming processors) , and the Tesla 1060 (240 streaming processors).
GPU-SPHysics is programmed in C++ and CUDA, the Nvidia extension to C, which provides the calls to the video card. CUDA is available for free from the Nvidia site: CUDA Zone.

GPU-SPHysics has two versions: 2D and 3D. The models include three different options for interpolation kernels: quadratic, cubic spline, and the Wendland kernel. Kernel correction for gradients is available if desired based on Chen et al. (1999), and Müller et al. (2003). XSPH (Monaghan, 1989) and Shepard and MLS filtering are also available.

Speedup

The speedup is calculated against the performance of the GPU code when run on a single CPU, an Intel Core2 Duo at 2.5GHz. (Note that this code is not optimal for CPU execution; it does not consider the symmetry in particle interactions, evaluating them twice (once per particle instead of once per particle pair). Although the independent, repeated evaluation is almost a necessity on the GPU, it is an unnecessary burden on the CPU; optimizing it away would improve the CPU speed by a factor of two, halving the speed-ups.)

The table below presents the speed-ups for the three major parts of the SPH algorithm : neighbor list construction, forces calculation and integration for the 3D dam break with 677000 particles.

GPU (# of processors)

8600m (32)

8800 GT (112)

GTX 280 (240)

neighbor list construction

4.4

11.8

15.1

forces calculation

28

120

207

integration

3.2

13.7

23.8

Considering that neighbor list construction, forces calculation and integration take respectively 56%, 42% and 2% of the calculation time we achieve a global speedup of 92 on a GTX 280.

3D Examples

Dam Break (or Bore in a Box)


Gómez-Gesteira and Dalrymple (2004) examined the "bore in a box" case, which was a dam break into a wet region with a structure, representing a tsunami attack on a building--shown in red below. The whole domain is surround by a wall. The GPU version of this case with over 100 times more particles (nearly five million) is more fully resolved showing the run-up jets in front of the obstacle and at the back wall corners.



The colors in the figure are associated with the magnitude of the velocity. The dam was located at the left of the figure (wall now removed). Note the bow wave around the square building in the middle of the box and the wake behind it. The dam break bore has reached the back wall and is beginning to fall back into the domain.

Waves in the Surf Zone

Breaking water waves are a particularly difficult problem, due to the nonlinearity of the waves, the splash-up and engendered turbulence. Dalrymple and Rogers (2008) used SPHysics to model waves in the surf zone. GPU-SPHysics has been applied to this problem using a Tesla card. This simulation, with over four million particles, provides a realistic looking surf zone, with three-dimensional coherent turbulent structures despite the two-dimensional wavemaker.

By increasing the width of the wavetank, we get to study features that occur on the open coast. For example, here is a tank as wide as it is long. Features such as wave setup on the shoreline is observed as well as some three-dimensionality in the waves (snapshot taken after about a dozen waves).

Levee Failure

The nature of the failure of a section of a levee, such as those on the Industrial Canal in New Orleans, govern the associated damage due to the impact of the flood waters. The short movie shows the slow failure of a wall section and the impact on flooding. Catastrophic wall failure (rapid falling of the wall) leads to much stronger impact forces as the water flows straight out from the levee rather than along the jetty as in this case. Download movie: WallFail6.m4v



The levee section is failing (note velocity magnitude is color coded); there is a splash off the lip of the falling wall, and water is flowing along the levee due to the slow nature of the breach.



Falling levee section is now overtopped and flood waters are flowing between the houses. Note the run-up in front of the houses due to the flow impact.

2D Examples

Falling Jets

Objects and bodies of fluid can be prescribed with initial velocities. On the left, below, is a snapshot of a 1.5 m long vertical "jet" with an imposed vertical velocity of -1.5 m/s taken 0.26 s after impact. Pinch-off (the closing of the penetration cavity by gravity) has not yet occurred.

After pinch-off, on the right, entrainment causes the falling jet to become unstable, as shown at 0.49 s. Note that the end of the jet has passed into the cavity and it is beginning to close.

 

Solidification of a lava lake

Comparison of our SPH results and semi-analytical solution (Manglik, 2005) for the cooling of a lava lake of 2km thickness at an initail temperature of 1100°C, a bottom temperature of  647°C and a top temperature of 0°C.

Position of solidification fronts (dot SPH, triangle semi-analytical solution)

Temperature profile at 0.004, 0.007 and 0.01 My (T in °C, solid line SPH, triangle semi-analytical solution) 

Solidification of a lava flow under radiative looses

Solidification of a 3m lava flow under radiative losses on top and conduction on ground.

Temperature profile at different times (T in °K, time in days)

Temperature evolution:



References

Chen, J.K., J.E. Braun, and T.C. Carney, A corrective Smoothed Particle Method for boundary value problems in heat conduction, International Journal of Numerical Methods in Engineering,, 45, 231-252, 1999.

Dalrymple, R.A. and B.D. Rogers, Numerical modeling of water waves with the SPH method,
Coastal Engineering, 53/2-3, 141-147, 2006.

Gómez-Gesteira, M. and R.A. Dalrymple, Using a 3D SPH method for wave impact on a tall structure,
Journal of Waterway, Port, Coastal, and Ocean Engineering, ASCE, 120, 2, 2003.

Hérault, A., presentation, SPHERIC 3rd Workshop, Lausanne, Switzerland, 2008.

Hérault, A., G. Bilotta, and R.A. Dalrymple, SPH on GPU with CUDA,
Journal of Hydraulic Research and Engineering, IAHR, sub judice.

Manglik, A., A moving boundary solution for solidification of a lava lake and magma intrusion in the presence of a time-varying contact temperature,
Journal of Earth System Science, 114, 2, 169-176, 2005.

Monaghan, J.J., On the problem of penetration in particle methods,
Journal of Computational Physics, 82, 1-15, 1989.

Monaghan, J.J., Simulating free surface flows with SPH,
Journal of Computational Physics, 110, 399-406, 1994.

Monaghan, J.J., Smoothed Particle Hydrodynamics,
Reports on Progress in Physics, 68, 1703-1759, 2005.

Müller, M., R. Keiser, A. Nealen, M. Pauley, M. Gross, and M. Alexa, Point based animation of elastic, plastic, and melting objects.
Eurographics/ACM SIGGRAPH Symposium on Computer Animation, 2004.