HiPerGator AI Hero-Run Feature

Title: A century of tradition replaced

Subtitle: Redefining 100 years of mathematical theory with HiPerGator

Porous materials, like a sponge or soil, have a connected pore space or cavities within them allowing fluids (liquid or gas) to flow through the system. But what happens when two fluids attempting to flow through the same porous material, such as oil and water, are immiscible? This scenario exists routinely in environmental, living, and engineered systems, and it presents unique challenges in fields such as hydrology, resource extraction, landscape conservation, global climate analysis, and cancer treatment. Because of the widespread nature of multiphase porous medium systems and their importance to society, accurate mathematical and computational models are critical.

One important consideration when developing these models is their scale and resolution. For example, imagine how complicated it would be to describe the systems from the scale of an entire landscape down to a clump of sand. This poses significant technological and resource constraints — both in physical representation, experimentation, and computation. Multiphase porous medium mathematical models could be based on a microscale approach in which the boundaries of all fluid and solid phases are accounted for. However, such approaches are not computationally feasible in practical applications. For instance, simulating a two-fluid flow on a supercomputer at the microscale through a representative volume of about 100 grains of sand can take more than 24 hours! Instead, a larger scale representation is often needed: the macroscale. The macroscale is a continuum scale in which points represent the averaged behavior of all phases and their boundaries.

In 1922, English meteorologist Lewis Fry Richardson published Weather Prediction by Numerical Process, which wrote down the first theoretical model for a macroscale two-fluid flow phenomenon through porous materials. It was a remarkable step toward developing a rigorous theory for its time when scientists did not have high-resolution imaging technology or high-performance computers. Richardson’s theoretical model, however, lacked precise definitions of quantities of physical scale, did not include interfacial quantities now known to be important, assumed a local equilibrium state, and did not account for the second law of thermodynamics (entropy production). The gaps in Richardson’s theory result in inaccuracies, and his theory cannot account for the vast variability in macroscale environments. Dr. Cass T. Miller and Dr. William G. Gray, professors at the University of North Carolina at Chapel Hill, sought to develop a more accurate model bridging across micro and macroscales. Their new theoretical framework, called Thermodynamically Constrained Averaging Theory (TCAT), addresses each of the deficiencies of Richardson’s model.

Developing a computational tool capable of capturing just a small range of the physical phenomena expressed in two-fluid flow is extremely challenging.  Over the last six years, Chris Fowler, a doctoral student at UNC-Chapell Hill and high performance computing (HPC)/artificial intelligence research scientist at Cambridge Computer, has worked to develop a lattice Boltzmann simulator — a numerical technique for the simulation of fluid flows — to support the Miller-Gray theory. The simulator, also known as the Lattice Boltzmann Porous Medium – University of North Carolina (LBPM-UNC) simulator, can capture the physics of fluid flow in porous systems at a highly resolved microscale on the largest supercomputers in the world. 

Chris Fowler Cambridge Computer

To demonstrate the readiness of the new simulator, he performed a hero-run on the University of Florida’s supercomputer – HiPerGator. With the support of UF Information Technology (UFIT) staff, UF Distinguished Professor and project owner Dr. Rafael Muñoz-Carpena, and UF’s NVIDIA AI Technology Center (NVAITC), Fowler used 512 GPUs across HiPerGator’s 64 DGX A100 SuperPOD nodes to benchmark and achieve a peak weak scaling performance of 90%. This means the computation achieved was 512 times larger than what would be possible on a single GPU and less than 10% efficiency was lost due to the necessary communications – a remarkable result.

The collaboration between Dr. Muñoz-Carpena, UFIT, and the NVAITC allowed the UNC team to work directly with HPC experts and cutting-edge supercomputing technologies. Mark Hill and Kristopher Keipert, two of NVIDIA’s senior solutions architects, provided insight and helped facilitate Fowler’s GPU analysis of the project. Once the simulator was ready to begin scaling up for a hero-run on HiPerGator, UFIT senior director Erik Deumens and his staff worked to scale the simulator from a single node to 64 nodes.  

Dr. Cass T. Miller University of North Carolina at Chapel Hill	Dr. Rafael Muñoz-Carpena University of Florida

“This collaboration across the disciplines and supported by unprecedented computational resources that UF spearheads is a wonderful example of how we need to advance the frontier of science and engineering,” said Dr. Muñoz-Carpena.

Thorough model validations are essential as 100 years of tradition are being replaced. By combining the theoretical and computational resources, this cross-university effort has advanced us closer to confirming the new theoretical model for two-fluid flow through porous materials. It will be essential to use high-performance computing resources and AI approaches to generate reliable parameter estimates to facilitate the application of the new TCAT modeling approach by those without HiPerGator-level computing facilities. Ultimately, the advances made in this work will underpin improved simulation of important processes such as carbon capture and storage, global climate trend prediction, environmental remediation, and tumor growth and treatment – all with much higher fidelity and accuracy than is currently available. Because the applications are so plentiful and important to society, this work is highly significant and broadly impactful.

“It is only through the joint use of theory, computation, and experimental approaches that such advancements are possible,” said Dr. Miller. “HiPerGator is one of the few machines in the world capable of simulating the scale of systems needed to enable these advancements.”