The TeraGrid is Transforming Research, Say Physicists at the American Physical Society Meeting in Pittsburgh
For academic researchers, these powerful resources are free, and though not all proposals get all they request, almost every proposal is granted access.
PITTSBURGH, March 30, 2009 – “high performance computing is transforming physics research,” said Ralph Roskies, co-scientific director of the Pittsburgh Supercomputing Center (PSC), during a presentation on Friday, March 20, at the American Physical Society Meeting, held in Pittsburgh, March 16-20.
“The Impact of NSF’s TeraGrid on Physics Research” was the topic of his talk, which led off a panel of physicists who have made major strides in their work through the TeraGrid, the National Science Foundation’s cyberinfrastructure program. “These world-class facilities,” said Roskies, “on a much larger scale than ever before, present major new opportunities for physics researchers to carry out computations that would have been infeasible just a few years ago.”
PSC Co-Director speaks at the American Physical Society March Meeting in Pittsburgh
To an audience of about 50, he emphasized that users do not pay to use the TeraGrid. The resources are provided by NSF and made available to researchers who write a proposal, which is reviewed by a committee drawn from the user community as a whole. While proposals often aren’t granted their full request of computational time, said Roskies, almost every proposal gets some access to TeraGrid, and it’s free.
He briefly explained how TeraGrid, a rich aggregation of diverse resources at multiple sites, nevertheless provides users with an integrated view – a single sign-on, single application form, single ticket system, coordinated user support, and simplified data movement that makes data sharing easy.
Roskies touched on TeraGrid’s diversity of resources, which now includes two powerful distributed-memory systems, Ranger at the Texas Advanced Computing Center (TACC) and Kraken at National Institute of Computational Science (NICS), University of Tennessee, each comprising more than 60,000 cores and providing capability in excess of 500 teraflops, with Kraken scheduled to go over a petaflop later this year. Shared-memory resources at the National Center for Supercomputing Applications (NCSA), University of Illinois, Champaign, Urbana (UIUC) and at PSC complement these systems, with tightly coupled clusters at several sites and a Condor pool at Purdue University that facilitates loosely coupled computations. Major resources on the way include a large shared-memory system at PSC, and Blue Waters, a 10-petaflop system at NCSA that will be available in 2011.
Before ceding the floor to four physicists whose work spans a range of physics domains – materials science (Axel Kohlmeyer, University of Pennsylvania); astrophysics (Tiziana Di Matteo, Carnegie Mellon University); quantum phenomena (Steven G. Johnson, MIT); and biophysics (Aleksei Aksimentiev, University of Illinois, Urbana-Champaign), Roskies highlighted several other examples of physics research on which the TeraGrid has had significant impact.
Simulations in quantum chromodynamics have used large allocations at several TeraGrid resource-provider sites, said Roskies, to study implications of the “standard model” of particle physics. Such simulations, he noted, are the only way to identify the experimental consequences of the theory. The computational complexity grows as one needs ever-smaller spacing on the lattices employed in the calculations (to approximate the continuum), larger lattices to encompass more space, and convergence with respect to quark masses, which are very small and still require extrapolations to compare physical and theoretical values.”Moreover, once you put in virtual quarks,” said Roskies, “the agreement between theory and experiment is wonderful, giving a lot of confidence that the simulations capture the essential physics.” Nevertheless, even with continual algorithmic improvements, virtual quarks greatly complicate the computations.
The results of these lattice computations, noted Roskies, are stored in an internationally agreed-upon format, so that physicists worldwide get the benefit of TeraGrid’s contribution in this field.
In astrophysics, Roskies cited Mike Norman’s group at the University of California, San Diego. The goal of their simulations is to evolve the cosmos from initial conditions and capture the physics of how very small spatial inhomogeneities (one part in 100,000) 380,000 years after the Big Bang get transformed by gravity into the severe inhomogeneities of today’s galaxies, stars and voids. Uniform meshes aren’t adequate to the task, and Norman’s team has used sophisticated adaptive-mesh techniques (seven levels of mesh refinement) to zoom in on dense regions where key physical processes occur. Because of the extreme challenges of load balancing, TeraGrid’s mix of large shared-memory’s for initial conditions and data analysis’ and powerful distributed memory systems, with fast data movement among systems and to storage, is essential. TeraGrid staff, said Roskies, has helped with major improvements in code efficiencies and visualization tools.
In materials science, Roskies described the TeraGrid’s NanoHUB Science Gateway developed at Purdue (in work led by Gerhard Klimeck), which provides an interface that non-experts can use for a set of modeling and simulation tools to address challenges of designing nanoscale components. In 2008, NanoHUB had more than 6,200 users, who ran more than 300,000 simulations, and supported 44 university classes.
In biophysics, Roskies described work by Klaus Schulten’s group at the University of Illinois, Urbana-Champaign on aquaporin, explaining how this channel protein’s ubiquitous in the body, conducts large volumes of water through cell walls while, at the same time, filtering out charged particles such as hydrogen ions (protons). Roskies showed a clip from Schulten’s animation, which was cited by the 2003 Nobel chemistry prize to Peter Agre for the structure of aquaporins, work that provided the starting place of Schulten’s simulations. This kind of simulation, molecular dynamics, noted Roskies, makes it possible to start from a known protein structure and see how that structure implies function.
Breaking down 2008 TeraGrid-enabled research by discipline, Roskies observed, shows that physics, including materials and astronomical sciences, comprised 39-percent of all usage. Add chemistry, and research in the physical sciences accounted for 52-percent of TeraGrid usage.
He cited rapid growth in TeraGrid usage, from 80-million to 300-million units in one year, with a near 50-percent increase in quantity of users, and he mentioned TeraGrid’s extensive training programs and its Campus Champions program, which facilitate access by entry-level computational researchers. “It’s free,” he iterated, “and we’ll even help you with coding and optimization. Think about problems you want to solve and talk to us. Don’t be constrained by what appears possible today.”