Identification of Cosmic-Ray Source by IceCube Neutrino Observatory Depended on Global Collaboration, PSC’s Bridges
Four billion years ago—before the first life had developed on Earth—a massive black hole shot a proton out at nearly the speed of light. A result of this cosmic catapult was the creation of a neutrino, a strange, tiny particle lighter than any other known type of matter.
Fast forward—way forward—to 45.5 million years ago. Early primates and elephants had just appeared on Earth. The Antarctic continent had started collecting an ice sheet. Eventually Antarctica would capture 61 percent of the fresh water on Earth, creating an ice mass of continental scale.
Thanks to PSC’s Bridges system and expert help from PSC staff, scientists running the IceCube Neutrino Observatory in Antarctica and their international partners have used these distant events as a tool to answer a hundred-year-old scientific mystery: Where do cosmic rays come from?
“The reason that IceCube was envisioned and built was to try and understand the origin of cosmic rays,” says Gonzalo Merino, computing facilities manager for the Wisconsin IceCube Particle Astrophysics Center at the University of Wisconsin-Madison.
IMAGE RIGHT: IceCube detector schematic: Detector By NASA-VERVE—IceCube Science Team—Francis Halzen, Department of Physics, University of Wisconsin
First identified in 1912, cosmic rays have puzzled scientists. The higher in the atmosphere you go, the more of them you can measure. The Earth’s thin shell of air, scientists came to realize, was protecting us from potentially harmful radiation that filled space. Today, the existence of cosmic rays remains a big concern in human space travel, posing a potential risk to astronauts. The concern is particularly high for long-term trips, such as to Mars.
“These particles that get to the Earth have an energy that is completely crazy,” Merino adds. “Orders of magnitude higher than anything we can produce with accelerators like the LHC,” the most powerful proton-smasher built by humans.
But where were cosmic rays coming from?
Making Straight the Path
Cosmic rays consist of electrically charged particles traveling at incredible speed. Most cosmic ray particles consist of a single proton. That’s the smallest positively charged particle of normal matter.
Cosmic ray particles are ridiculously powerful. Merino compares the force of a proton accelerated by the man-made LHC as similar to the force of a mosquito flying into a person. By comparison, the “Oh-My-God” cosmic ray particle detected by the University of Utah in 1991 hit with the force of a baseball flying at 58 miles per hour! Theorists posed the idea that cosmic rays come at us with so much energy—in other words, at such a high speed—because a powerful cosmic engine out in space somewhere had accelerated them, in a kind of cosmic-scale electric engine.
But scientists realized they had a problem in identifying the source of cosmic rays. Because these particles are electrically charged, they would be pushed and pulled by every magnetic field they encounter along the way. Cosmic rays would move through space like pinballs bouncing off of a series of bumpers. They would not travel in a straight line. You can’t figure out where they originated from their direction when they hit Earth.
Particle-physics theorists came to the rescue.
“If cosmic rays hit any matter around them, the collision will generate secondary products,” Merino says. “A byproduct of any high-energy interaction with the protons that make up much of a cosmic ray will be neutrinos.”
Neutrinos are very odd. A weird, under-weight cousin of ordinary matter, they have less than a millionth of the mass of a tiny electron. Neutrinos respond to gravity and to what’s known as the “weak subatomic force.” But they aren’t affected by the electromagnetic forces that send cosmic rays on a drunkard’s walk. Scientists realized that the intense showers of protons at the source of cosmic rays had to be hitting matter nearby. This would create a spray of high-energy neutrinos as a byproduct. Cosmic rays won’t travel in a straight line through the universe. But the neutrinos that they produce can. If the scientists could detect these high-energy neutrinos coming from a massive powerful object in space, that object was likely creating cosmic rays.
“If we could identify the source of such high-energy neutrinos, we could identify with certainty that protons were being accelerated and colliding with matter,” Merino explains.
The Shape of Water
The neutrino’s virtue in pinpointing cosmic-ray-generating objects—the fact that magnetic fields or matter won’t affect their path—was a curse as well as a blessing. If the matter that makes up your instrument can’t interact with an incoming neutrino, how are you going to detect it?
The answer lay in neutrinos’ ability to interact with matter via the weak atomic force. This is the force that causes radioactive decay.
Many neutrinos will pass through normal matter without giving a trace of their presence—by some estimates, about 100 billion of the lower-energy neutrinos generated by the sun pass through the tip of your finger every second. But every once in a long while, one will hit the nucleus of an atom of that matter. This causes a shower of other particles that instruments can detect. Such collisions will be rare. But they will happen.
“The probability that a neutrino will interact with matter is extremely low, but not zero,” Merino explains. “If you want to see neutrinos, you need to build a huge detector so that they collide with matter at a reasonable rate.”
Enter the Antarctic ice shelf. It would be impractical or impossible to build a detector with enough mass to capture a significant number of cosmic neutrinos. But nature has already done the job. The ice in the Antarctic ice shelf, itself nearly pure water, could be used as a detector. From 2005 through 2010, a University of Wisconsin-Madison-led team created the IceCube Neutrino Observatory by drilling 86 holes deep in the ice, dangling electronic detectors from wires into the holes, and then letting the holes re-fill and re-freeze. They then had an instrument that consisted of 5,160 detectors suspended in a huge ice cube six-tenths of a mile on each side and weighing a little over a billion tons.
In IceCube, the scientists had created a new kind of telescope. By tracing the path of the particles created by a neutrino hitting the ice backwards, they could assume the direction that the neutrino had been traveling. And that path, again followed backward, would point straight to the neutrino’s birthplace.
How’s the Water?
The IceCube scientists weren’t quite ready to detect cosmic-ray-associated neutrinos. Theory had given them a good idea of what properties such neutrinos should have. But while the IceCube observatory was nearly pure water, it wasn’t completely pure. It would contain trace contaminants. As a natural formation, its transparency might differ a bit from spot to spot. The scientists needed to figure out how these imperfections would affect the light they needed to detect.
Glass sphere: An IceCube digital optical module (DOM) “taklampa” to be deployed in hole #85. IceCube drill camp, Dec. 13, 2009
They began with computer simulations of the light traveling through ice using traditional supercomputers containing central processing units (CPUs). These are the standard processors that carry out calculations on general-purpose computers. The scientists realized quickly, though, that portions of their computations would work more quickly on a newer type of processor called a graphics-processing unit (GPU). Invented to help video games have more realistic animation, GPUs have since found application in a large number of scientific computations.
“Understanding the optical properties of this hunk of ice was a key point for improving the resolution of our instruments,” Merino says. “We realized that a part of the simulation is a very good match for GPUs. These computations run 100 to 300 times faster on GPUs than on CPUs. In 2012, we started our quest of finding ever-increasing GPU time.”
Madison’s own GPU cluster—a supercomputer made up of many processors, in this case 400 of them—proved useful. But it was too small to handle all the work. Collaborators offered time on their own campuses’ GPU systems. It was never enough.
Then Merino had a talk with PSC’s Sergiu Sanielevici, who filled him in on Bridges’ large GPU capability, as well as that on PSC partner institutions in the NSF’s XSEDE supercomputing network.
“We did a small-scale test of GPUs on XSEDE machines and very quickly convinced ourselves that we would be able to access these GPU resources in an effective way,” Merino says. “XSEDE then became an important addition to our GPU capacity … the more GPUs we have, the more we can improve the detector’s resolution.”
The IceCube scientists could not afford to assume that running their computer code in Bridges would be easy. They had created their own massive and complex flow of calculations. These could have slowed down considerably, or even come to a jarring halt, had the new machines conflicted with it.
“Normally our GPU simulations are one step inside a pretty long simulation chain,” Merino explains. “Our computing framework involves an enormous number of jobs” running on a large network of computers. “Bridges and the other XSEDE GPU resources integrated seamlessly; that was very important for us in the sense that we were able to use the resources very quickly without any re-engineering of our framework.”
One of the attractive points specifically about Bridges, Merino explains, is the availability of the 32 advanced NVIDIA P100 GPU nodes on the system. “Bridges rolled out the P100 very early, which allowed us to do benchmark testing … The capability to access a large number of P100s was a huge advantage. The effective power of Bridges is very nice in that regard.”
Paydirt
Once the Wisconsin-led scientists had built their ice telescope and convinced themselves they knew how it would respond to cosmic-ray neutrinos, they had to wait for one to come along. Again, many of these particles would pass through IceCube’s huge detector block. But very few would hit an atom in an ice molecule and produce the spray of particles they were looking for.
On Sept. 22, 2017, it happened. A single high-energy neutrino created a path of light through the detector array. An automated system tuned to the signature of a cosmic-ray neutrino sent a message to the scientists in Madison. The automated alarm also gave collaborators running observatories around the world and in space a similar heads-up.
The collaborators were important because a single neutrino detection would not have been proof by itself. Theory had predicted that the cosmic-ray particle collision that created the neutrino would also bring into being radio waves and gamma radiation. It was important for scientists at observatories that detect those types of radiation to look at the same spot in the sky.
The collaborators found multiple types of radiation coming from the same spot in the sky as the neutrino. At this spot was a “blazar” called TXS 0506+056, about 4 billion light years from Earth. Blazars are a type of active galactic nucleus (AGN). That’s a huge black hole sitting in the center of a distant galaxy that is flaring as it eats the matter in the galaxy surrounding it. What makes an AGN a blazar is that, by chance, the jet of radiation coming from the AGN is pointed straight at us.
The collaborators had collected a number of firsts. The first detection of a cosmic-energy neutrino pointing to an active astronomical object. The first coordination of observatories of multiple types of radiation—“multimessenger observation”—to pinpoint and verify a source of cosmic rays. Multimessenger astrophysics has been a major, new NSF thrust for answering questions about the Universe.
The scientists think that the vast forces surrounding the black hole are likely the catapult that shot cosmic-ray particles on their way toward Earth, as well as caused a shower of neutrinos and other radiation in the material in the blazar’s galaxy. After a journey of 4 billion years across the vastness of space, one of those neutrinos blazed a path through IceCube’s detector.
The IceCube scientists weren’t finished yet. They went back over nine and a half years of previous detector data, before the automated system was warning them of cosmic-energy neutrinos. They found that their telescope had been detecting neutrinos from TXS 0506+056 all along. The added detections from this analysis greatly raised their confidence in their initial detection.
The two findings—the multimessenger observation and the analysis of past detections—led to papers in the prestigious journal Science in July 2018. Future work, including more detailed simulations run on the increasing number of new-generation GPUs in the XSEDE system, will focus on confirming that blazars are the source—or at least a major source—of the high-energy particles that fill the Universe.
Captions:
IceCube detector schematic:
Detector By NASA-VERVE—IceCube Science Team—Francis Halzen, Department of Physics, University of Wisconsin
South pole image:
Ice Cube drilling setup at drill camp, December 2009
Glass sphere:
An IceCube digital optical module (DOM) “taklampa” to be deployed in hole #85. IceCube drill camp, Dec. 13, 2009