Making Big Data DANCE(S)
XSEDE Project Successfully Tests Scheduled Networking of Big Data
July 22, 2016
A project to make movement of the largest datasets more efficient has tested for the first time networking hardware components necessary for scheduling network bandwidth using Software Defined Networking (SDN). Researchers from the National Science Foundation’s XSEDE collaboration of supercomputing sites reported at the XSEDE16 conference in Miami how they tested transfer of Big Data between two XSEDE sites, the Pittsburgh Supercomputing Center (PSC) and the National Institute for Computational Sciences (NICS) at the University of Tennessee.
“If you think about [high performance computing] users and their end-to-end workflow, they need to do it by breaking their projects into pieces,” says Victor Hazlewood of NICS, principal author of the peer-reviewed article accompanying the presentation. “One of these pieces inevitably involves moving data into and out of where researchers do their computation.”
Unfortunately for those moving the largest Big Data sets, Hazlewood adds, today’s research wide-area networks (WAN) are more a free-for-all than their compute and storage counterparts, which are scheduled and managed. By design, the protocol software that regulates the network data flow treats all data packets as equal. When the networks are not crowded it’s like an empty freeway. When they are crowded, though, the situation becomes much like a convoy of buses getting broken apart by other traffic. For large datasets traveling across the WAN, that can mean slow movement or even failures to move the data completely in a reasonable amount of time.
The NSF-funded DANCES (Developing Applications with Networking Capabilities via End-to-End SDN) is researching a kind of on-demand high-occupancy vehicle lane for large data transfers across the WAN, allowing Big Data users to specify how much bandwidth they would like to use and schedule their job so that they can be assured the data will get through in a timely fashion.
The paper presented at XSEDE16 describes the team’s survey of network switching hardware that met the DANCES OpenFlow feature requirements. The paper also describes the team’s successful tests using the DANCES research environment in moving large data between PSC and NICS.
“We were … interested in devices that supported bandwidth reservation,” says coauthor Kathy Benninger of PSC. “Part of that has to do with adoption of the newest OpenFlow versions by switch vendors.”
OpenFlow 1.3, the latest commonly deployed version of the standard network communications interface, has features that support such bandwidth management. But the problem, Benninger explains, is that “compliance” with a given version of OpenFlow doesn’t require a particular network switch to support every facet of that version’s specification. The researchers had to test a number of switch products, eventually choosing the Corsa DP6410 as most fully compatible with the DANCES requirements, successfully testing the combination in a number of data-transfer scenarios between PSC and NICS. Next, the group plans to test the full set of end-to-end system functions that will be needed to support real data transfers by XSEDE users.
Hazlewood expects that the SDN technology employed by DANCES may be adopted by major coordinated HPC networks such as XSEDE within the next five years, with more mainstream use in five to 10 years.
The XSEDE16 conference showcases the discoveries, innovations, challenges and achievements of those who use and support XSEDE resources and services, as well as other digital resources and services throughout the world. The theme of XSEDE16 is “Diversity, Big Data & Science at Scale: Enabling the Next-Generation of Science and Technology.”
Developing Applications with Networking Capabilities via End-to-End SDN (DANCES)