History of the High Performance Computing Collaboratory

Lineage

The High Performance Computing Collaboratory (HPC²) at Mississippi State University originated in 1990 as the NSF Engineering Research Center (ERC) for Computational Field Simulation at MSU, which focused directly on the application of high performance computing (HPC) for computational field simulation of fluid flow, heat and mass transfer, and structural mechanics for applications to aircraft, spacecraft, ships, automobiles, environmental, ocean, and biological flow problems. Initially funded by the NSF Engineering Research Center (ERC) Program in 1990 -- one of three NSF ERCs funded that year out of 48 proposals - this ERC was the only one of the NSF ERCs with its focus directly on high performance computing (HPC). Over the 11-year life cycle that NSF ERCs have, the Center increased its annual funding by an order of magnitude, graduating from the NSF ERC program in 2001 and now continuing as a self-sufficient research center with funding from a range of federal agencies and industry. The NSF ERC at MSU was cited by the NSF Director in the January 1999 issue of ASEE Prism as a prime example of a successful NSF ERC, noting that the NSF ERC at MSU "effectively demonstrates that you can institute change in a very positive way".

MSU's successful proposal in the highly competitive NSF ERC Program was enabled by strategic directions taken earlier in the Department of Aerospace Engineering at MSU to place emphasis on computational fluid dynamics (CFD) research and in the Department of Electrical and Computer Engineering to emphasize microelectronics research. CFD research in the ASE Department attained national recognition in the 1980s, and joined with established microelectronics research in the ECE Department in a DARPA-funded research effort to tailor HPC for CFD applications. This cross-disciplinary effort provided the foundation for the successful NSF ERC proposal that was submitted in 1989.

The intent of the NSF ERC Program is to change the culture of the university in the direction of multidisciplinary research and collaboration, and the NSF ERC at MSU has indeed brought major change to MSU, establishing a pattern of cross-disciplinary research effort - an acceptance of such effort as the norm - that has positioned the University for other major opportunities. The NSF ERC also established a new multidisciplinary graduate program in Computational Engineering cutting across engineering, computer science, and mathematics that allows entry from any field of engineering or even from the physical and life sciences.

The impact of the NSF ERC on undergraduate education at MSU also has been significant, not only in the enhancement of course offerings through new faculty, new technology, and new course content, but also through the research experiences provided for undergraduate students and the student jobs involving directly relevant work experience. Nearly 600 undergraduate students, and about that many graduate students, have shared in the NSF ERC cross-disciplinary experience.

In addition to research efforts over the years, the NSF ERC constructed an interactive computer simulation display entitled "How Wings Work" for the "How Things Fly" gallery in the National Air and Space Museum of the Smithsonian Institution on the Mall in Washington DC. And the NSF ERC designed the logo and handled all the graphics for Supercomputing'94 held in Washington DC in 1994. Working with the Department of Art and the School of Architecture at MSU, the NSF ERC facilitated the new graduate degrees in animation and electronic visualization (MFA in Art) and in electronic design (MS in Architecture).

The NSF ERC for Computational Field Simulation

As created in the NSF ERC Program, the NSF ERC for Computational Field Simulation at MSU was a multidisciplinary academic research center conducting a coordinated research program according to a strategic plan to advance the U.S. capability in the use of computational simulation in engineering analysis and design, as well as in scientific research in general. This NSF ERC focused on all elements involved in the computational simulation of physical field phenomena - physical processes occurring over space and time, i.e. governed by partial differential equations: computationally intense simulations requiring access and efficient utilization of HPC facilities at the highest level, and requiring distributed graphics at the highest level for effective collaboration with other centers of effort.

The NSF ERC incorporated engineers, physicists, computer scientists, and mathematicians in cross-disciplinary research in geometrical representation,numerical solutions, and scientific visualization -- together with the underlying parallel computing environments and mathematical foundations. Although the Center's historical concentration was in computational fluid dynamics (CFD), its strategic research efforts in building computational problem solving environments encompassed all areas of field physics. The Center's effort in CFD spanned the gamut from high-speed aircraft to ships and submarines to ocean modeling and on to biological systems.

As an NSF ERC, the ERC for Computational Field Simulation at Mississippi State had the mission of interacting with industry and federal labs in research of importance to economic competitiveness and national security. The specific mission of this NSF ERC was to develop high-level capability for computational field simulation of physical problems for application in analysis and design. That mission was approached through research thrusts in five areas: grid generation, solution algorithms, scientific visualization, system software, and computer architecture - mounting an integrated research program in both software and hardware.

A computational field simulation system requires geometrical representation, numerical solutions, and scientific visualization operating in a coordinated environment making efficient and effective use of parallel computing platforms friendly to the user. The NSF ERC strategically addressed all these elements in such combination and with an applications focus.

This NSF ERC built and expanded on established nationally recognized research effort in grid generation at Mississippi State (recognized by the1992 AIAA Aerodynamics Award), and made major advances in unstructured grid generation, as well as in its traditional area of block-structured grid generation. The NSF ERC produced the comprehensive Handbook of Grid Generation published by CRC Press in 1999.

The NSF ERC's long-standing nationally-recognized research effort in computational fluid dynamics (CFD), with real-world applications on complex engineering problems using both block-structured and unstructured grids, positioned the Center to immediately bring state-of-the-art CFD capability to bear in collaboration with other universities through this connection. This CFD effort of the NSF ERC was specifically cited by the Navy for on-time delivery, has provided the basis of turbomachinery simulation capability made available to industry by NASA, and was a major component of the Navy maneuvering submarine simulation effort. This established concerted capability in geometry/solution/visualization applications served well for expansion into all areas of field physics through collaborations with other universities.

In the parallel computing environment area, the NSF ERC was a leader in the development and deployment of the Message Passing Interface (MPI) standard enabling distributed computing, and the adoption of the Center's implementation of MPI by several HPC hardware vendors was noted in HPCwire. The Center also co-authored a primary text on MPI.

Original Mission

The fields addressed by the Center are simply regions or volumes of space within which physical phenomena vary with position and time. These physical phenomena impact our lives much more than we generally realize: Common examples of these phenomena include compressible fluid flow, such as the air flow around aircraft and automobiles; incompressible fluid flow, such as the flow of water past ships; and electromagnetic (EM) fields, such as microwave signal transmission or local EM fields around power lines. Historically, extensive experiments and costly equipment, such as wind tunnels, were necessary to study these phenomena; however, they can now be simulated using modern powerful computing systems and computational techniques. This use of field simulations has become a powerful tool to supplement and/or replace traditional experimental and analytical methods. In some instances, simulation is the only approach for understanding immeasurable physical phenomena or analyzing particular designs under certain conditions.

The primary obstacle to widespread use of computational field simulation (CFS) by industry has been that industry's design-related field problems are usually quite complex, requiring simulations that are time-consuming and costly. The NSF Engineering Research Center (ERC) for Computational Field Simulation at Mississippi State University was created in 1990 with the mission to research the means and methods to reduce the time and cost while increasing the fidelity and scope of complex field simulations for engineering analysis and design. When the ERC was created, complete real-world problems were impractical to simulate on that generation of supercomputers. Capturing the complex geometry of a complete aircraft and creating the discrete small-volumes in the regions for field computations could easily have taken 6-12 months with extensive engineering efforts. Since the field computations for the total 3-D problem could easily have required another 6-12 months on a $20 million supercomputer, the complete problems were too expensive for practical simulations.

The Center currently conducts coordinated cross-disciplinary research (which amounts to approximately $12 million per year) interacting with 16 industrial affiliates and 14 government affiliates. The Center's vision is to enable for U.S. industry and government agencies superior capabilities for computational field simulations of large-scale geometrically complex physical field problems through domain-specific integrated simulation environments for rapid analysis and design, facilitating a shift from physical prototyping towards computational simulation prototyping. The research in the ERC focuses on the underlying science of CFS and the development of means and methodologies to enable the necessary reduction in engineering time, clock time, and overall cost of CFS for application domains, including extensions into diverse, very complex multidisciplinary applications, that are relevant to industry. A major emphasis of the Center--which employs 70 faculty and staff researchers and approximately 125 students from various disciplines in engineering, science, and mathematics--has been in computational fluid dynamics (CFD) to provide the means to simulate complete real world applications (such as cruise missiles, complete submarines with rotating propulsors, biofluid flow with particulates, rocket exhaust, and weapon or stage separation). However, the Center's strategic research efforts in building computational problem-solving environments encompass all areas of field physics.

The fulfillment of the Center's mission is illustrated by the John Glenn space shuttle flight. The Center has significantly contributed to the art and practice of "unstructured grid generation", yielding high quality grids in significantly less time. Whereas "structured grid generation" on a total aircraft may take several weeks or months, the Center's unstructured grid generation can be accomplished within a day. The Center focused a team on coupling its structured grid CFD algorithm knowledge within a portable, scalable computational architecture onto unstructured grid solver technology. This required substantial research in both boundary layer gridding and solution algorithms. As it turned out, the parallel solver (research) code had just been assembled for the first time when the Space Shuttle mission STS-95 was launched. NASA Johnson Space Center called seeking simulated analysis of the Space Shuttle Orbiter during the return flight after the Orbiter drag chute door was lost during main engine startup. [The NASA engineers wanted to know the dynamic pressure in the region of the missing chute door in order to estimate the aerodynamic loadings during reentry.] The ERC group read a previously supplied Space Shuttle Orbiter geometry into the ERC's integrated simulation environment (SOLSTICE) and created the grids within hours. Initial simulation results were computed on a high performance computer within two days. The significance of this endeavor was not that NASA actually needed the results for successful reentry, but rather that the ERC had been able to take a tough real world problem and compute the solutions in two to three days after receiving the geometry description. [The turnaround time could have been reduced to a day if the ERC's main high-performance computer was dedicated solely to this task (only 1/4 of the machine was actually used)]. This demonstrated an achievement that was a direct result of the ERC's mission and efforts. Our researchers have demonstrated superior ability to simulate very complex real world problems with complex geometries in relative motion. These accomplishments have come from directed cross-disciplinary efforts involving various technologies: grid generation, field solution algorithms, and scientific visualization, coupled with computer and computational engineering. Without the ERC structure, we could not have combined all of the various talents and technologies required.

The simulation of field phenomena is historically divided into three phases: grid generation (i.e., capturing a representation of the geometry and field regions and then constructing a grid that divides these regions into many separate or discrete small volumes); the use of solution algorithms to solve discrete approximations for the equations which govern the physical phenomena (constitutive equation modeling) to obtain values for the physical solution at each point in space and time; and scientific visualization (i.e., displaying the geometry and/or solution on a computer screen). Further, the computational capability itself is enabled by the computing system, including the system software, which creates the application programming support environment, and the computer architecture, which incorporates the hardware features and constraints.

In addition to the achievements in unstructured grid generation, the ERC researchers pioneered the development of structured grid generation and techniques based on the parametric-based nonuniform rational B-splines (NURBS) representation. These technologies are incorporated into the block-structured grid and unstructured grid tools: generalized unstructured multi-block GUM-B and HyperMesh, respectively.

The ERC originated, released, and supported a code to simulate turbomachinery-related flows--a code that has become the defacto standard for turbomachinery manufacturers which are served by NASA Lewis Research Center (Allied Signal, Allison, General Electric, and Pratt & Whitney). Algorithms enabling simulations for a fully configured submarine or surface ship, including rotating propulsors have also significantly advanced the state-of-the-art with the Navy selecting the codes for technology transfer to the shipyards. Other examples of application-enabling research include (1) capabilities for free-surface (air/water) boundaries (e.g., ships and littoral water oceanography with actual geometry, temperature, and salinity), (2) both chemical and thermal nonequilibrium flow (e.g., the Space Shuttle main engine nozzle starting transient; the Titan Centaur booster and separation for Lockheed Martin; an environmental quality network model for underground pollution; a fully two-dimensional radiative heat transfer model; a portable, scalable solver for arbitrary mixtures of thermally perfect gases in local chemical equilibrium; and preconditioning algorithms for low-speed combustion applications), and (3) particle-laden biofluid flows (e.g., first ever oscillating flow in a pulmonary bifurcation section, inhaled aerosols through branching, lung-like tubes; and powder-carrying air jets for industrial coating processes). These efforts clearly demonstrate the ERC’s expertise in creating efficient means to simulate fluid flow through moving complex geometries with complex physical phenomena. The scope of CFS capabilities is being extended across new single and multidisciplinary domains.

In scientific visualization the ERC has contributed by (1) advancing the technology in visualizing time-varying data through the release of ISTV in February 1996 and by (2) demonstrating at SuperComputing '95 the ability to steer and visualize a running ocean model in a multiperson immersive environment. Current research encompasses feature detection, multiresolution visualization, data compression, distributed visualization, flow visualization, and interactive virtual environments, exemplified by the recent installation of a CAVE.

A close association between computing and field simulation researchers within the Center has resulted in efficient parallel CFS algorithms. Simulations of both compressible and incompressible flows have been researched to develop effective solution methodologies and programming environments for creating portable parallel simulation programs for use with the evolving computer architectures. The ERC has been a leader in the emergence of the Message Passing Interface (MPI) as the standard paradigm for writing distributed (parallel) applications, enabling programs to be portable across a wide range of distributed computing platforms. (MPI parallel applications can be ported across shared or distributed memory architectures "transparently" while exploiting the low latency of shared memory.) A significant event in technology transfer has been the collaboration with computer companies to expedite deployment of MPI software on various platforms.

The Center is actively involved in evolving a testbed CFS integrated system, incorporating the various elements in an effective and user-oriented system and focused on next generation computational simulations. The CFS testbed provides for capability demonstrations, a modular framework for technology advancements and maintenance, efficient reuse across physics domains, vehicles for technology transfer, and tools for CFS instruction. Targeted for collaborative use, the integrated system testbed provides the foundation for creating domain-specific versions, such as an integrated simulation system for littoral waters, for affiliates. Technology transfer is facilitated through collaborative research activities, focusing on the particular customer and industrial needs. Current ERC research is leading to addition of the integration of measurements and to the addition of multidisciplinary simulations to the capability of the simulation testbed.

As part of its education mission, the ERC has had approximately 700 students directly involved in the research of the Center, has developed a cross-disciplinary computational engineering graduate program to allow students to integrate their study with the research of the Center, has developed graduate and undergraduate CFS courses for engineering students and others, and has developed a minor in computational engineering for undergraduate engineering students. Working with the Department of Art and the School of Architecture, the ERC facilitated the new graduate degrees in animation and electronic visualization (MS in art) and in electronic design (MS in architecture). The ERC has programs with minority and women’s institutions and works with K-12 schools.