Skip to main content

Advertisement

SpringerLink
Log in

A parallel finite element method for two-phase flow processes in porous media: OpenGeoSys with PETSc

  • Original Article
  • Published:
Environmental Earth Sciences Aims and scope Submit manuscript

Abstract

In past decades, high performance computing has became a valuable tool in many fields of environmental science and technology to utilize computational power for better characterization of the complexity of environmental systems as well as predicting their evolution in time. In this work, a parallel computing technique is presented for the numerical simulation of two-phase flow processes in porous media. The Galerkin finite element method (FEM) is used to solve the initial boundary value problem arising from the underlying mathematical model. The PETSc package is utilized for parallelization of the computational task in both the global assembly of the system of linear equations and the linear solver. In order to parallelize the global assembly of the linear equation system, the overlapping domain decomposition method is used. The preset parallel FEM approach is realized within the framework of OpenGeoSys, an open source C++ finite element code for numerical simulation of thermal, hydraulic, mechanical and chemical processes in fractured porous media. The computational efficiency of the approach has been tested with three examples of increasing complexity, the five spot benchmark, dense non-aquaeous phase liquid infiltration into a inhomogeneous porous medium and a real-world application to the \(\mathrm {CO}_2\) storage research site: Ketzin, in Germany.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  • Abou-Rjeili A, Karypis G (2006) Multilevel algorithms for partitioning power-law graphs. In: Proceedings of the 20th international conference on parallel and distributed processing (IPDPS’06). IEEE Computer Society, Washington, DC. http://dl.acm.org/citation.cfm?id=1898953.1899055

  • Anderson E, Bai Z, Bischof C, Blackford S, Demmel J, Dongarra J, Du Croz J, Greenbaum A, Hammarling S, McKenney A, Sorensen D (1999) LAPACK users’ guide, 3rd edn. Society for Industrial and Applied Mathematics, Philadelphia

    Book  Google Scholar 

  • Aziz K, Settari A (1979) Petroleum reservoir simulation, vol 476. Applied Science Publishers, London

    Google Scholar 

  • Balay S, Brown J, Buschelman K, Eijkhout V, Gropp W, Kaushik D, Knepley MG, McInnes LC, Smith BF, Zhang H (2013) PETSc users manual. Tech. Rep. ANL-95/11-Revision 3.4, Argonne National Laboratory

  • Bastian P (1999) Numerical computation of multiphase flow in porous media. Habilitationsschrift, Universität Kiel, Kiel

    Google Scholar 

  • Bastian P, Ippisch O, Rezanezhad F, Vogel HJ, Roth K (2007) Numerical simulation and experimental studies of unsaturated water flow in heterogeneous systems. In: Reactive flows, diffusion and transport. Springer, New York, pp 579–597

  • Bauer S, Beyer C, Kolditz O (2006) Assessing measurement uncertainty of first-order degradation rates in heterogeneous aquifers. Water Resour Res 42(1):W01420

  • Bauer S, Class H, Ebert M, Feeser V, Götze H, Holzheid A, Kolditz O, Rosenbaum S, Rabbel W, Schäfer D, Dahmke A (2012) Modeling, parameterization and evaluation of monitoring methods for CO\(_2\) storage in deep saline formations: the CO2-MoPa project. Environ Earth Sci 67(2):351–367

    Article  Google Scholar 

  • Böttcher N, Taron J, Kolditz O, Park C-H, Liedl R (2012) Evaluation of thermal equations of state for CO\(_2\) in numerical simulations. Environ Earth Sci 67(2):481–495

    Article  Google Scholar 

  • Brunner P, Simmons CT (2012) HydroGeoSphere: a fully integrated, physically based hydrological model. Ground Water 50(2):170–176

    Article  Google Scholar 

  • Delfs J-O, Park C-H, Kolditz O (2009) A sensitivity analysis of hortonian flow. Adv Water Resour 32(9):1386–1395

    Article  Google Scholar 

  • Farhat C, Roux F (1991) A method of finite element tearing and interconnecting and its parallel solution algorithm. Int J Numer Methods Eng 32(6):1205–1227

    Article  Google Scholar 

  • Farhat C, Roux F (1992) An unconventional domain decomposition method for an efficient parallel solution of large-scale finite element systems. SIAM J Sci Stat Comput 13(1):379–396

    Article  Google Scholar 

  • Flemisch B, Darcis M, Erbertseder K, Faigle B, Lauser A, Mosthaf K, Müthing S, Nuske P, Tatomir A, Wolff M et al (2011) Dumu\(^x\): dune for multi-\(\{\)phase, component, scale, physics, \(\ldots\) \(\}\) flow and transport in porous media. Adv Water Resour 34(9):1102–1112

    Article  Google Scholar 

  • Forum TM (1993) MPI: a message passing interface

  • Fujisawa T, Inaba M, Yagawa G (2003) Parallel computing of high-speed compressible flows using a node-based finite-element method. Int J Numer Methods Eng 58(3):481–511

    Article  Google Scholar 

  • Hairer E, Wanner G (1996) Solving ordinary differential equations II: stiff and differential-algebraic problems. In: Springer series in computational mathematics, 2nd revised edn, vol 14. Spinger, Berlin

  • Hammond GE, Lichtner PC (2010) Field-scale model for the natural attenuation of uranium at the Hanford 300 Area using highperformance computing. Water Resour Res 46(W09527)

  • Hammond GE, Lichtner PC, Rockhold ML (2011) Stochastic simulation of uranium migration at the Hanford 300 Area. J Contam Hydrol 120:115–128

  • Hammond GE, Lichtner PC, Rockhold ML (2011) Stochastic simulation of uranium migration at the Hanford 300 Area. J Contam Hydrol 120121:115–128

    Article  Google Scholar 

  • Helmig R (1997) Multiphase flow and transport processes in the subsurface: a contribution to the modeling of hydrosystems. Springer, New York

    Book  Google Scholar 

  • Heroux M, Bartlett R, Hoekstra VHR, Hu J, Kolda T, Lehoucq R, Long K, Pawlowski R, Phipps E, Salinger A, Thornquist H, Tuminaro R, Willenbring J, Williams A (2003) An overview of Trilinos. Tech. Rep. SAND2003-2927, Sandia National Laboratories

  • Hubschwerlen N, Zhang K, Mayer G, Roger J, Vialay B (2012) Using TOUGH2-MP on a cluster-optimization methodology and study of scalability. Comput Geosci 45:26–35

    Article  Google Scholar 

  • Kempka T, Class H, Görke UJ, Norden B, Kolditz O, Kühn M, Walter L, Wang W, Zehner B (2013) A dynamic flow simulation code intercomparison based on the revised static model of the Ketzin pilot site. Energy Procedia (EGU GA) 40:418–427

    Article  Google Scholar 

  • Kempka T, Kühn M (2013) Numerical simulations of CO\(_2\) arrival times and reservoir pressure coincide with observations from the Ketzin pilot site, Germany. Environ Earth Sci 70(8):3675–3685

    Article  Google Scholar 

  • Kolditz O, Bauer S, Beyer C, Böttcher N, Dietrich P, Görke U-J, Kalbacher T, Park C-H, Sauer U, Schütze C, Shao H, Singh A, Taron J, Wang W, Watanabe N (2012a) A systematic benchmarking approach for geologic CO\(_2\) injection and storage. Environ Earth Sci 67(2):613–632

    Article  Google Scholar 

  • Kolditz O, Bauer S, Bilke L, Böttcher N, Delfs J, Fischer T, Görke U, Kalbacher T, Kosakowski G, McDermott C et al (2012b) OpenGeoSys: an open-source initiative for numerical simulation of thermo-hydro-mechanical/chemical (THM/C) processes in porous media. Environ Earth Sci 67(2):589–599

    Article  Google Scholar 

  • Kolditz O, Görke U, Shao H, Wang W (eds) (2012c) Thermo-hydro-mechanical-chemical processes in fractured porous media: benchmarks and examples. In: Lecture notes in computational science and engineering, vol 86. Springer, New York

  • Kolditz O, de Jonge J (2004) Non-isothermal two-phase flow in low-permeable porous media. Comput Mech 33(5):345–364

    Article  Google Scholar 

  • Lacroix S, Vassilevski YV, Wheeler MF (2001) Decoupling preconditioners in the implicit parallel accurate reservoir simulator (ipars). Numer Linear Algebra Appl 8(8):537–549

    Article  Google Scholar 

  • Law K (1986) A parallel finite element solution method. Comput Struct 23(6):845–858

    Article  Google Scholar 

  • Lewis RW, Schrefler BA (1998) The finite element method in the static and dynamic deformation and consolidation of porous media, 2nd edn. Wiley, New York

    Google Scholar 

  • Lichtner PC (2001) FLOTRAN users guide: two-phase nonisothermal coupled thermal hydrologic chemical (THC) reactive flow and transport code. Tech. Rep. LA-UR-012348, Los Alamos Natl. Lab

  • Lichtner PC, Hammond GE (2012) Using high performance computing to understand roles of labile and nonlabile U(VI) on Hanford 300 Area plume longevity. Vadoze Zone J 120121:115128

    Google Scholar 

  • Maroto-Valer MM, Song C, Soong SY (2002) Environmental challenges and greenhouse gas control for fossil fuel utilization in the 21st century. Springer, New York

    Book  Google Scholar 

  • Martens S, Kempka T, Liebscher A, Lüth S, Möller F et al (2012) Europe’s longest-operating on-shore CO\(_2\) storage site at Ketzin, Germany: a progress report after three years of injection. Environ Earth Sci 67(2):323–334

    Article  Google Scholar 

  • McDermott C, Randriamanjatosoa A, Tenzer H, Kolditz O (2006) Simulation of heat extraction from crystalline rocks: the influence of coupled processes on differential reservoir cooling. Geothermics 35(3):321–344

    Article  Google Scholar 

  • Metz B, Davidson O, De Coninck H, Loos M, Meyer L (2005) IPCC special report on carbon dioxide capture and storage. Tech. rep., Intergovernmental Panel on Climate Change, Geneva. Working Group III

  • Mills RT, Lu C, Lichtner PC, Hammond GE (2007) Simulating subsurface flow and transport on ultrascale computers using pflotran. In: Journal of physics: conference series, vol 78. IOP Publishing, p 012051

  • Mills RT, Sripathi V, Mahinthakumar G, Hammond GE, Lichtner PC, Smith BF (2010) Engineering pflotran for scalable performance on cray xt and ibm bluegene architectures. In: SciDAC 2010 annual meeting

  • Mykkeltvedt TS, Nordbotten JM (2012) Estimating effective rates of convective mixing from commercial-scale injection. Environ Earth Sci 67(2):527–535

    Article  Google Scholar 

  • Norden B, Frykman P (2013) Geological modeling of the Stuttgart formation at Ketzin, Germany. Int J Greenh Gas Control (in press). doi:10.1016/j.ijggc.2013.04.019

  • Park C-H, Böttcher N, Wang W, Kolditz O (2011) Are upwind techniques in multi-phase flow models necessary? J Comput Phys 230(22):8304–8312

    Article  Google Scholar 

  • Pau G, Bell J, Pruess K, Almgren A, Lijewski M, Zhang K (2010) High-resolution simulation and characterization of density-driven flow in CO\(_2\) storage in saline aquifers. Adv Water Resour 33(4):443–455

    Article  Google Scholar 

  • Person M, Banerjee A, Rupp J, Medina C, Lichtner P, Gable C, Pawar R, Celia M, McIntosh J, Bense V (2010) Assessment of basin-scale hydrologic impacts of CO\(_2\) sequestration, Illinois basin. Int J Greenh Gas Control 4(5):840–854

    Article  Google Scholar 

  • Salinger AG, Xiao Q, Zhou Y, Derby JJ (1994) Massively parallel finite element computations of three-dimensional, time-dependent, incompressible flows in materials processing systems. Comput Methods Appl Mech Eng 119(1–2):139–156

    Article  Google Scholar 

  • Sanavia L, Pesavento F, Schrefler BA (2006) Finite element analysis of non-isothermal multiphase geomaterials with application to strain localization simulation. Comput Mech 37(4):331–348

    Article  Google Scholar 

  • Schrefler BA, Matteazzi R, Gawin D, Wang X (2000) Two parallel computing methods for coupled thermohydromechanical problems. Comput-Aided Civ Infrastruct Eng 15(3):176–188

    Article  Google Scholar 

  • Shao H, Dmytrieva S, Kolditz O, Kulik D, Pfingsten W, Kosakowski G (2009) Modeling reactive transport in non-ideal aqueous-solid solution system. Appl Geochem 24(7):1287–1300

    Article  Google Scholar 

  • Shioya R, Yagawa G (2005) Large-scale parallel finite-element analysis using the internet: a performance study. Int J Numer Methods Eng 63(2):218–230

    Article  Google Scholar 

  • Singh A, Goerke U-J, Kolditz O (2011) Numerical simulation of non-isothermal compositional gas flow: application to carbon dioxide injection into gas reservoirs. Energy 36(5):3446–3458

    Article  Google Scholar 

  • Tezduyar TE, Sameh A (2006) Parallel finite element computations in fluid mechanics. Comput Methods Appl Mech Eng 195(13–16):1872–1884

    Article  Google Scholar 

  • Topping BHV, Khan AI (1996) Parallel finite element computations. Saxe-Coburg Publications, Edinburgh

    Google Scholar 

  • Toselli A, Widlund O (2004) Domain decomposition methods-algorithms and theory, vol 34. Springer, New York

    Google Scholar 

  • Tuminaro R, Heroux M, Hutchinson S, Shadid J (1999) Official Aztec user’s guide: version 2.1. http://www.sandia.gov/~rstumin/aztecguide.ps

  • Wang W, Kolditz O (2007) Object-oriented finite element analysis of thermo-hydro-mechanical (THM) problems in porous media. Int J Numer Methods Eng 69(1):162–201

    Article  Google Scholar 

  • Wang W, Kolditz O (2010) Sparse matrix and solver objects for parallel finite element simulation of multi-field problems. In: Zhang W, Chen Z, Douglas CC, Tong W (eds) High performance computing and applications, vol 5938., Lecture Notes in Computer Sciences. Springer, Berlin, pp 418–425

    Chapter  Google Scholar 

  • Wang W, Kosakowski G, Kolditz O (2009) A parallel finite element scheme for thermo-hydro-mechanical (thm) coupled problems in porous media. Comput Geosci 35(8):1631–1641

    Article  Google Scholar 

  • Wang W, Schnicke T, Kolditz O (2011) Parallel finite element method and time stepping control for non-isothermal poro-elastic problems. Comput Mater Contin 21(3):217–235

    Google Scholar 

  • Watanabe N, Wang W, McDermott C, Taniguchi T, Kolditz O (2010) Uncertainty analysis of thermo-hydro-mechanical coupled processes in heterogeneous porous media. Comput Mech 45(4):263–280

    Article  Google Scholar 

  • Zhang K, Croisé J, Mayer G (2011) Computation of the couplex-gaz exercise with TOUGH2-MP: hydrogen flow and transport in the pore water of a low-permeability clay rock hosting a nuclear waste repository. Nucl Technol 174(3):364–374

    Google Scholar 

  • Zhang K, Wu Y-S, Pruess K et al (2008) Users guide for TOUGH2-MP-a massively parallel version of the TOUGH2 code. Report LBNL-315E, Lawrence Berkeley National Laboratory, Berkeley

Download references

Acknowledgments

This work is partially supported by the CO2MAN project funded by the German Federal Ministry of Education and Research (BMBF), and the industrial partners of VNG, Vattenfall, RWE, Statoil, Dillinger Hüttenwerke, Saarstahl and OMV. The authors would like to express their thanks to Thomas Kempka and Sonja Martens at Helmholtz-Centre Potsdam-GFZ, German Research Centre for Geosciences for their constructive comments. The authors are also grateful to the vital support of the Linux cluster team at UFZ, Thomas Schnicke, Ben Langenberg and Christian Krause. In addition, the efforts by the developers of PETSc are very much appreciated. Those open source projects are extremely useful for scientific code development in general. We also thank Leslie Jakobs at UFZ for proofreading the manuscript.

Author information

Authors and Affiliations

  1. Helmholtz Center for Environmental Research-UFZ, 04318 , Leipzig, Germany

    Wenqing Wang, Thomas Fischer, Norbert Böttcher, Uwe-Jens Görke & Olaf Kolditz

  2. Federal Institute for Geosciences and Natural Resources, Berlin, Germany

    Björn Zehner

  3. Dresden University of Technology, Dresden, Germany

    Olaf Kolditz

Authors
  1. Wenqing Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thomas Fischer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Björn Zehner
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Norbert Böttcher
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Uwe-Jens Görke
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Olaf Kolditz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wenqing Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, W., Fischer, T., Zehner, B. et al. A parallel finite element method for two-phase flow processes in porous media: OpenGeoSys with PETSc. Environ Earth Sci 73, 2269–2285 (2015). https://doi.org/10.1007/s12665-014-3576-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12665-014-3576-z

Keywords

Search

Navigation