Faculty Spotlights

HiPerGator AI Hero-Run Feature

Title: A century of tradition replaced

Subtitle: Redefining 100 years of mathematical theory with HiPerGator

Porous materials, like a sponge or soil, have a connected pore space or cavities within them allowing fluids (liquid or gas) to flow through the system. But what happens when two fluids attempting to flow through the same porous material, such as oil and water, are immiscible? This scenario exists routinely in environmental, living, and engineered systems, and it presents unique challenges in fields such as hydrology, resource extraction, landscape conservation, global climate analysis, and cancer treatment. Because of the widespread nature of multiphase porous medium systems and their importance to society, accurate mathematical and computational models are critical.

One important consideration when developing these models is their scale and resolution. For example, imagine how complicated it would be to describe the systems from the scale of an entire landscape down to a clump of sand. This poses significant technological and resource constraints — both in physical representation, experimentation, and computation. Multiphase porous medium mathematical models could be based on a microscale approach in which the boundaries of all fluid and solid phases are accounted for. However, such approaches are not computationally feasible in practical applications. For instance, simulating a two-fluid flow on a supercomputer at the microscale through a representative volume of about 100 grains of sand can take more than 24 hours! Instead, a larger scale representation is often needed: the macroscale. The macroscale is a continuum scale in which points represent the averaged behavior of all phases and their boundaries.

In 1922, English meteorologist Lewis Fry Richardson published Weather Prediction by Numerical Process, which wrote down the first theoretical model for a macroscale two-fluid flow phenomenon through porous materials. It was a remarkable step toward developing a rigorous theory for its time when scientists did not have high-resolution imaging technology or high-performance computers. Richardson’s theoretical model, however, lacked precise definitions of quantities of physical scale, did not include interfacial quantities now known to be important, assumed a local equilibrium state, and did not account for the second law of thermodynamics (entropy production). The gaps in Richardson’s theory result in inaccuracies, and his theory cannot account for the vast variability in macroscale environments. Dr. Cass T. Miller and Dr. William G. Gray, professors at the University of North Carolina at Chapel Hill, sought to develop a more accurate model bridging across micro and macroscales. Their new theoretical framework, called Thermodynamically Constrained Averaging Theory (TCAT), addresses each of the deficiencies of Richardson’s model.

Developing a computational tool capable of capturing just a small range of the physical phenomena expressed in two-fluid flow is extremely challenging. Over the last six years, Chris Fowler, a doctoral student at UNC-Chapell Hill and high performance computing (HPC)/artificial intelligence research scientist at Cambridge Computer, has worked to develop a lattice Boltzmann simulator — a numerical technique for the simulation of fluid flows — to support the Miller-Gray theory. The simulator, also known as the Lattice Boltzmann Porous Medium – University of North Carolina (LBPM-UNC) simulator, can capture the physics of fluid flow in porous systems at a highly resolved microscale on the largest supercomputers in the world.

Chris Fowler
Cambridge Computer

To demonstrate the readiness of the new simulator, he performed a hero-run on the University of Florida’s supercomputer – HiPerGator. With the support of UF Information Technology (UFIT) staff, UF Distinguished Professor and project owner Dr. Rafael Muñoz-Carpena, and UF’s NVIDIA AI Technology Center (NVAITC), Fowler used 512 GPUs across HiPerGator’s 64 DGX A100 SuperPOD nodes to benchmark and achieve a peak weak scaling performance of 90%. This means the computation achieved was 512 times larger than what would be possible on a single GPU and less than 10% efficiency was lost due to the necessary communications – a remarkable result.

The collaboration between Dr. Muñoz-Carpena, UFIT, and the NVAITC allowed the UNC team to work directly with HPC experts and cutting-edge supercomputing technologies. Mark Hill and Kristopher Keipert, two of NVIDIA’s senior solutions architects, provided insight and helped facilitate Fowler’s GPU analysis of the project. Once the simulator was ready to begin scaling up for a hero-run on HiPerGator, UFIT senior director Erik Deumens and his staff worked to scale the simulator from a single node to 64 nodes.


Dr. Cass T. Miller University of North Carolina at Chapel Hill	Dr. Rafael Muñoz-Carpena University of Florida

“This collaboration across the disciplines and supported by unprecedented computational resources that UF spearheads is a wonderful example of how we need to advance the frontier of science and engineering,” said Dr. Muñoz-Carpena.

Thorough model validations are essential as 100 years of tradition are being replaced. By combining the theoretical and computational resources, this cross-university effort has advanced us closer to confirming the new theoretical model for two-fluid flow through porous materials. It will be essential to use high-performance computing resources and AI approaches to generate reliable parameter estimates to facilitate the application of the new TCAT modeling approach by those without HiPerGator-level computing facilities. Ultimately, the advances made in this work will underpin improved simulation of important processes such as carbon capture and storage, global climate trend prediction, environmental remediation, and tumor growth and treatment – all with much higher fidelity and accuracy than is currently available. Because the applications are so plentiful and important to society, this work is highly significant and broadly impactful.

“It is only through the joint use of theory, computation, and experimental approaches that such advancements are possible,” said Dr. Miller. “HiPerGator is one of the few machines in the world capable of simulating the scale of systems needed to enable these advancements.”

The Molecules of Life

Researchers Adrian Roitberg and Jinze Xue Basic biology textbooks will tell you: all life on Earth is built from four types of molecules. The four molecules of life are proteins, carbohydrates, lipids, and nucleic acids, with each of the four groups vital for every single living organism.

Until a decade ago, conducting research on the evolution and interactions of large collections of atoms and molecules could only be modeled using very simple computer simulation experiments, because the computing power needed to handle the large data-sets to understand life at its most basic form just wasn’t available.

Experiment harnessed the full power of the HiPerGator supercomputer over the 2023 Winter Break.

Basic biology textbooks will tell you: all life on Earth is built from four types of molecules. The four molecules of life are proteins, carbohydrates, lipids, and nucleic acids, with each of the four groups vital for every single living organism.

During UF’s 2023 Winter Break, Jinze Xue, a Ph.D. student in the Roitberg Computational Chemistry Group, conducted a large-scale early Earth chemistry experiment utilizing more than 1000 A100 GPUs on HiPerGator. A molecular dynamics experiment was performed on 22 million atoms that identified 12 amino acids, three nucleobases, one fatty acid, and two dipeptides. The discovery of larger molecules, which would not have been possible in smaller computing systems, is a significant achievement.

Dr. Adrian Roitberg, V.T. and Louise Jackson Professor in UF’s Department of Chemistry, and his research group have been exploring the use of Machine Learning (ML) to study chemical reactions for the past six years.

“Our previous success enabled us to use ML/AI to calculate energies and forces on molecular systems, with results that are identical to those of high-level quantum chemistry--but around one million times faster!

The accomplishments empowered us to ask the following question: Can one create ‘molecules of life’, such as amino acids and DNA bases starting from a simple mix of gases and simulating their reactions?

Roitberg continues:

“These questions have been asked before, but due to computational limitations, previous calculations used small numbers of atoms and could not explore the range of time needed to obtain results. But with HiPerGator, we can do it!”

Erik Deumens, senior director in UFIT for Research Computing, explains how this full takeover of HiPerGator was possible:

"HiPerGator is heavily used throughout the year for research and education. But it has the unique capability to run very large `hero’ calculations that use the entire machine with the potential to lead to breakthroughs in science and scholarship. Such calculations require careful scheduling to not impact the normal business of supporting research and teaching, which is why we do them around quieter times, like Christmas and over the July 4^th holidays. When we found out about the work Dr. Roitberg’s group was doing, we approached him to try a `hero’ run with the code he developed.”

The emergence of AI and very powerful GPUs can enable such highly compute and data intensive scientific simulations to be carried out--calculations scientists could only imagine just a few years ago. The combination of AI models that speed up the dynamics of very large collections of atoms and molecules coupled with the power of the University of Florida’s supercomputer (the fastest in U.S. higher education) enabled the experiment to be done.

Roitberg continues:

“Using Machine Learning methods, we created a simulation using the complete HiPerGator set of GPUs. We were able to see, in real time, the formations of almost every amino acid (e.g., alanine, glycine, etc.) and a number of very complex molecules. This was very exciting to experience.”

This project is part of the effort to discover how complex molecules can form from basic building blocks, and to make the process automatic through large computer simulations. The molecules of life is one example of this general idea.

Roitberg notes that he and his research group spent many hours working with members of UFIT. Ms. Ying Zhang, UFIT’s AI Support Manager, ran point for this experiment.

“Ying put together a team comprised of Research Computing staff and staff from NVIDIA to help scale compute runs, provide invaluable advice and help, and accelerate analysis of the data to the point where the analyses were done in just seven hours, instead of the three days we initially expected it to take. We met every week from initial conception to the final results, in a very fruitful collaboration.”

The results, and the short time in which HiPerGator was able to deliver them is inspiring new lines of inquiry from Roitberg and others who are getting closer every day to answering questions about how complex molecules, such as those seen in life, is formed.

“This is a great opportunity for UF faculty,” says Roitberg. “Having HiPerGator in-house, with the incredible staff willing to go above and beyond to help researchers produce ground-breaking science like this, is something that makes my non-UF colleagues very jealous.”

HiPerGator, the University of Florida supercomputer, went online in 2013 with 16,000 cores. It has been expanded and upgraded every few years since its initial go-live. HiPerGator’s last big expansion with the latest technologies was in 2020 and put into production in 2021. HiPerGator is housed in the UF Data Center, a 25,000+ sq. ft, Leadership in Energy and Environmental Design (LEED®) certified building. Learn more about HiPerGator: https://rc.ufl.edu/get-started/hipergator/.

Dr. Yonghui Wu

Medical AI tool from UF and NVIDIA gets human thumbs-up in first study

A new artificial intelligence computer program created by researchers at UF and NVIDIA can generate doctors’ notes so well that two physicians couldn’t tell the difference, according to an early study from both groups. A new artificial intelligence computer program created by researchers at UF and NVIDIA can generate doctors’ notes so well that two physicians couldn’t tell the difference, according to an early study from both groups.

Dr. Mildred Maldonado-Molina

Enabling research @ UF Family Data Center

Billions of rows of data that is subject to specific protections under federal and state laws…UF’s Institute for Child Health Policy & the Family Data Center leverages this incredible data volume, often referred to as “Big Data”, to investigate ways to improve healthcare delivery in Florida. But first, staff must ensure the records they are entrusted with are safe from exposure.

Dr. Mildred Maldonado-Molina is leading the efforts at the Family Data Center. She worked closely with UFIT to develop a solution that works for research while also compliant with all information security and privacy laws. A scientist, she and her team did not have the time or expertise to build the necessary computing infrastructure. Roland Estrella, manager of clinical research for the Department of Health Outcomes and Biomedical Informatics, summarizes what was built:

“Working with UFIT, the Family Data Center developed a robust computing environment. We have on-demand access to data from multiple state agencies and the capacity to process billions of medical and administrative records under strict federal information security guidelines.”

Dr. Stephen Coombes

^{Identifying damage in the brain’s superhighway}

UF researchers have developed a template showing the brain’s superhighways and how they are impacted by a stroke. The brain images required to create the template were processed on HiPerGator.

“We’re interested in the structure of the brain after a stroke,” said Stephen Coombes, assistant professor of Applied Physiology and Kinesiology. “Collecting and analyzing images of brains from people that haven’t experienced a stroke helps us track the different motor pathways in the brain; sort of a ‘Google Maps’ for the brain’s corticospinal tract.”

The benefits of mapping the corticospinal tract — it’s a superhighway for movement — can have a significant impact to the care and recovery of stroke patients.

“Knowing which part of the tract is damaged may be extremely helpful in predicting recovery after stroke,” notes Coombes. “Physical therapists can also use this information to prescribe more individualized rehabilitation exercises.”

Utilizing 3,000 HiPerGator cores, Coombes said Dr. Derek Archer’s imaging needs were completed in three months. Without HiPerGator’s processing power, analyzing the data on a single computer would have taken an astounding 42 years of processing time. For more information about Coombes’s and Archer’s work on the corticospinal tract template and its applied use capabilities, visit the Laboratory for Rehabilitation Neuroscience web page.

_{DR. MELANIE CORRELL}

^{Bringing genetic information into crop models}

_{AARON BEVERIDGE}

^{Using HiPerGator to work with social media big data}

_{DR. FORREST MASTERS}

^{Reducing loss of life and property during extreme wind events}

_{DR. JULIE JOHNSON}

^{Understanding sources of variability in how people respond to medications}

_{DR. DAMON WOODARD}

^{Using biological signals to establish individual identity}