Web100: Facilitating Effective and Transparent Network Use

3/13/00

While the national high-performance network infrastructure has grown tremendously both in bandwidth and accessibility, it is still common for applications, hosts, researchers and other users to be unable to take full advantage of this new and improved infrastructure. Without expert attention from network engineers, users are unlikely to achieve even 10 Mbps single stream TCP transfers, despite the fact that the underlying network infrastructure can support data rates of 100Mbps or more. On unloaded networks, this poor performance can be attributed primarily to two factors: host system software (principally TCP) that is optimized for low bandwidth environments and the lack of effective instrumentation and tools to diagnose performance issues at the end hosts.

The goal of the Web100 project is to enable ordinary network users to attain full network data rates without help from network experts. Currently, partial solutions to these host-based TCP problems are understood and, in many cases, prototype solutions exist in the network research community. However, these solutions have not been deployed in a comprehensive and systematic fashion. Issues include: insufficient attention to deployment; operating system and commercial software vendors that focus on the relatively low performance consumer market; the fact that each vendor chooses to install different improvements; and finally, that TCP/IP is designed in such a way as to hide all underlying performance problems. Consequently, current vendor software fails to attain appropriate data rates without help from network experts.

The overall goal of Web100 is to develop the components necessary for a high-performance network host-software environment. This environment will run common TCP-based applications at the fastest rate appropriate for the underlying network infrastructure. We will achieve TCP performance-tuning transparency in part by embedding the appropriate diagnostics and automatic controls in the end systems operating software at a level invisible to the user. Such an approach will have a two-fold benefit: it will free the end user from the need to have detailed knowledge of the network in order to use it effectively, and it will automate the network tuning that network engineers currently do by hand.

For the Web100 project to be successful, early adoption by both the user community and the vendor community is essential. We will address this by developing and implementing a user support and deployment infrastructure for the Web100 suite. Initially, this component will focus on understanding the needs of a well-defined user community and making sure these needs are addressed in the software suite. Support will later be expanded to include direct technical help to early adopters of the software suite. We will develop relationships with the vendor community at this point in the project as well, beginning with computer and operating system vendors, including those with direct interest in the Linux operating system, which will be used for our initial developments. In order to facilitate adoption by the commercial community, we will use the IETF standards process to standardize any modifications to the TCP-MIB.

This project is a collaboration of the Pittsburgh Supercomputing Center (PSC), the National Center for Atmospheric Research (NCAR), and the National Center for Supercomputing Applications (NCSA). Together these three centers will provide the technical expertise and user support necessary to develop, deploy, and support the Web100 suite. This effort builds on the joint experience of these organizations in providing engineering and application support in the area of high-performance networking and applications as part of the National Laboratory for Applied Network Research (NLANR).

2. Technical Description

The Web100 project can be broken down into three distinct areas: Core Technologies, User Support Services, and Vendor Liaison. While the design and development of core Web100 technologies is the initial focus of the project, developing User Support Services and establishing relationships with the commercial vendor community are also vital to the overall plan. As we have seen in the past, new technologies can often fail if they are not easy to use, do not have adequate support, do not address the needs of the end user community, or are not adopted into commercial products. Thus, User Support Services are essential, not only to identify end user needs and help educate and assist the end user, but also to provide a feedback path to the developers so that the end product actually meets the needs of the users. Similarly, technologies that are not adopted by the commercial vendor community will, at best, be used by only a select, technically savvy, subset of the community. By establishing links to the commercial sector, utilizing the IETF standards process, and developing commercial vendor partnerships, the Web100 project will develop a broader user base. Detailed descriptions of all three components of the project are discussed below.

2.1 Core Technology

In order to focus the project and ensure that its benefits will be available to a large portion of the user community, we have identified several key components essential to an environment capable of supporting high data transfers for a substantial subset of the high-performance networking community. These include a computer development platform, operating system instrumentation technologies, applications, and a packaging and distribution mechanism.

2.1.1 Computer Platforms

During the first year of the project, Intel-based Linux will be the principal development platform. This platform is readily available and widely used by the scientific community. It uses an open source code based operating system, is cost effective, and supports a large range of network-based applications. During subsequent years of the project, additional platforms will be added to the Intel-based platform. However, additional platforms will be supported only in conjunction with partnerships developed as a result of the Vendor Liaison component of this project.

2.1.2 Operating System Instrumentation Technologies

The starting point for the development of the Web100 software suite will be to endow TCP with better instrumentation. This instrumentation is the foundation for both the TCP autotuning performed in process-level code, as well as the process-level tools designed to locate bottlenecks within the following major subsystems: the sending application, the sending OS, the Internet path, the receiving OS, and the receiving application. As part of the applications work, measurement tools will also be built on this instrumentation to display performance indicators to end-users, as well as provide internal diagnostics for network and system administrators.

Specifically, we will design a management information base (MIB) for TCP that will replace or expand the existing TCP-MIB (RFC2012). The current TCP-MIB, written at a time when the IETF mandated a MIB for all protocols, instruments only a small subset of TCP parameters. It contains a total of only 19 variables, which are focused primarily on global statistics, including total number of connections, total data segments and total errors. Similarly, the per-connection table only indicates connected status. Given the limited nature of this TCP-MIB, it has never been widely implemented or used. Furthermore, nothing in the current TCP-MIB gives any indication of the cause of the bottleneck, i.e., what is controlling or limiting TCP performance.

Therefore, a new TCP-MIB will be the cornerstone of improved TCP instrumentation, providing a simple but elegant means of understanding the underlying operation of TCP within a host. Initially the TCP-MIB, as well as supporting instrumentation, will be designed to reliably distinguish between the following bottlenecks: TCP receiver (e.g., data limited by receiver window), IP path (e.g., the congestion window) and TCP sender (e.g., data availability at the sender). Once the prototype TCP-MIB is designed, developed and tested, we will develop additional TCP-MIB based monitoring tools to instrument other aspects of TCP performance.

In addition to information about TCP protocol events (counts of lost packets, timeouts, recoveries, etc.), the new path portion of the TCP-MIB will include metrics specific to analytical performance models [MSMO99]. These will parallel the Bulk Transport Metrics under development in the IETF IPPM working group [RFC2330, MA99, and Mat99]. The new TCP-MIB will also include some read-write variables which will allow user-space processes to adjust specific TCP tuning parameters, such as the limits on TCP buffer space. These variables will permit TCP autotuning to be split between the kernel and user mode.

2.1.3 Applications

2.1.3.1 Autotuning

A crucial initial use for the TCP-MIB and its prototype Linux kernel implementation will be to provide a means for performing dynamic, transparent, automatic TCP tuning from a user level process. Autotuning, the ability to automatically tune TCP to simultaneously achieve maximum throughput across all connections for all applications within the resource limits of the host, has already been demonstrated and published for NetBSD [SMM98].

Web100 will expand this work by utilizing the TCP-MIB to partition autotuning between the kernel and user mode and to make a number of extensions and improvements to the autotuning control algorithms. Internet path properties will be dynamically read by the autotuning daemon and used to update TCP tuning parameters such as buffer limits on the TCP retransmit and re-assembly queues. The autotuning daemon will be responsible for budgeting resources across all connections to maximize overall network performance without depleting system resources. There are significant advantages to splitting the autotuning function between the kernel and user mode: it greatly reduces the risk involved in deployment by minimizing changes to core TCP code; it simplifies the testing of multiple algorithms to balance resource needs and limits; and it permits complex policy mechanisms to constrain autotuning. A user-level autotuning daemon can also implement more complex tuning policies than would be practical to implement in the kernel.

A key Linux kernel implementor indicated that autotuning as published [SMM98] could not be included in a production Linux distribution due to the risks associated with possible autotuning behaviors in known hostile environments. All of these risks can be overcome by partitioning autotuning in such a way that the kernel implements only key TCP tuning adjustments and performance instruments (via the MIB), while all of the complex and subtle control algorithms run in user mode where they can be easily extended, replaced or disabled.

During the first year of the project, a prototype autotuning daemon will be implemented. During the second and third years, it will be improved and extended to include policy control mechanisms. It is expected that third parties will develop their own extensions to the autotuning daemon and that Web100 will be able to take advantage of this work by incorporating it into its own code. The third year of the project will focus on propagating autotuning to additional operating systems

2.1.3.2 Tools for Performance Measurement

Using information available from the TCP-MIB, a suite of host-based network diagnostic and performance measurement tools will be developed which will give the user and network administrator a dynamic view of the behavior of individual TCP sessions. These tools will be controlled through an intuitive graphical user interface and will span a spectrum of possible detail, from simple tools for users, to more complex and sophisticated tools for engineers. One such tool will be able to graphically and concurrently display as a function of real-time any selected subset of TCP-MIB variables. All tools will include capture and playback capabilities so that anomalous events can be captured and analyzed later. A tool library will be developed so that all tools will be compatible with respect to capture, playback and TCP-MIB variable selection and display options.

Since the first year of the project will focus primarily on TCP-MIB and associated Linux TCP kernel code development, only rudimentary tools will be developed during this time. The second year of the project will see intense effort focused on development of the tool library and an initial set of diagnostic and performance measurement tools. More sophisticated tools will be developed during the third year, using feedback from users and engineers to guide this effort. We also expect that numerous third parties will develop their own diagnostic and performance tools, which Web100 will be able to incorporate it into its own code and/or provide multiple alternatives in planned Web100 distributions.

2.1.3.3 FTP

The FTP program is one of the most important applications for moving bulk scientific data over the Internet. The July 1999 report from the Advanced Networking Infrastructure Needs in the Atmospheric and Related Sciences (ANINARS) Workshop, states:

This same report continues:

Web100 can most quickly address the known needs of the scientific community by implementing an optimized FTP by leveraging ongoing application tuning work at NCSA. Such an FTP would statically estimate the delay-bandwidth product from the round-trip time and the interface speed. It would use the existing socket API to adjust TCP tuning parameters to support the estimated delay bandwidth product. While there are some limitations to this approach, including the inability to make mid-transfer tuning adjustments, it would provide immediate relief to a very serious problem for the scientific community.

This optimized FTP will be implemented and distributed as quickly as possible during the first year of the project. It will be available to the user community before the TCP-MIB. While the initial implementation will be for the Linux platform, it can easily be ported to other operating systems. Users who are using operating systems other than Linux can continue to use the optimized FTP until autotuning is available for their systems. The quick distribution of this application will help to develop, assess and evaluate the Web100 user support organization. Finally, the optimized FTP will be used as a testing vehicle for prototype versions of the TCP-MIB by providing an additional mechanism for verifying TCP-MIB based instruments. Similarly, it will be used to help test and verify the autotuning implementation.

In order to make this distribution of FTP most useful, and recognizing the requirements for greater security in networked applications, the optimized FTP will support state-of-the-art authentication mechanisms. A significant portion of the target audience will be users of high-performance computing centers, most of which are now requiring stronger authentication mechanisms (e.g. Kerberos, ssh, PKI, etc). Providing support for these authentication mechanisms will make the FTP software useful in moving data to and from important resources to the research community.

2.1.4 Packaging and Distribution

By the end of the first year of the project, code will be available for distribution via downloads from a Web100 web site. Early in the second year of the project, Web100 will produce a complete CD-based Linux distribution. This distribution will likely be based on a popular Linux distribution such as Red Hat, Caldera, Mandrake or Debian; the Web100 kernel patch set, libraries and tools will be added to the selected base distribution. During the third year of the project, as part of the Vendor Liaison work, Web100 will offer its kernel changes for addition to the standard Linux kernel distribution and attempt to have the other parts of its distribution adopted by one or more commercial Linux distributors.

2.2 User Support

One of the primary objectives of Web100 is to produce a software suite that provides not only the desired performance, but also truly meets the needs of the target user community. We have learned through the NLANR project that new technologies are often initially difficult to use and successfully integrate into an existing application environment. By establishing User Support Services early in the project cycle, we intend to reduce the time it takes to get the Web100 suite adopted by the end user community. We envision three critical components to user support:

• Working with the user community and providing feedback to the developers;
• Providing support to facilitate the integration of the Web100 suite into the user’s application environment; and
• Providing documentation and training to the user community.

During the first year of the project we will focus on the identification and understanding of end-user requirements associated with effective utilization of the high-performance network infrastructure in the scientific community. We will survey the user community to determine their overall needs and requirements. Survey results will be used to help focus the development of the Web100 technologies. For example, although Linux has been chosen as the initial development platform, we need to understand what other platforms are critical to the user community and develop a strategy for migrating the Web100 technologies to these platforms. By identifying key requirements and incorporating them into the design of the Web100 software suite, we improve the chance that this software will be adopted by end users and incorporated into their research applications.

Also during Year 1, User Support Services will identify and begin working with early adopters of the technology. We will select a diverse group, including users who are technically proficient as well as those with little or no direct network experience. We will work with the early adopters to integrate the Web100 software into both their operating system and application environments. When necessary, User Support Services will help re-build end-host platforms, recompile application software and document application performance both before and afterwards. Based on our experience with the early adopter community, we will develop concise, readable and “user friendly” documentation for the initial Web100 distribution.

By the beginning of Year 2, we will develop and implement an overall support infrastructure for the broader scientific community. This infrastructure will also be based on our experience with the early adopter community. At minimum, it will:

• provide a mechanism for distribution of the Web100 suite to the community;
• help integrate Web100 into the end systems and applications;
• understand what does and does not work;
• provide feedback to the developers; and
• develop plans for continued support after the grant period ends.

The support infrastructure will include a central help desk that will provide quick access to the Web100 distribution and documentation as well as provide answers to basic technical questions. The second component will be focused User Support staff who will work directly with specific user communities and groups. This support will be done in a number of ways, including on-site support for specific sites or disciplinary groups. Focused support staff will act as a bridge between the users and the developers, providing bug reports and general feedback to the development team. This information will then be used to refine and adapt the Web100 software suite to meet the needs of the users. The end result of this close link between the user community and the development team will be a final Web100 distribution that can be easily used by the general scientific community and the Internet community as a whole. Another result will be the development of a model for distributed Focus User Support Teams that will provide tailored support services for specific discipline groups or campuses while keeping a communication channel open with the primary Web100 production group.

During Year 2, we will begin developing training materials and offering training courses to the user community. As before, we will initially target the early adopter community and then expand the audience over time. Training courses will be offered on a regular basis in a number of forums. We will leverage existing disciplinary conferences and workshops as well as provide site-specific and group specific training sessions. When possible, training materials will be available on the WWW server to provide an on-line training facility for Web100 users.

In Year 3, we will continue to provide support for new tools and utilities to the early adopter community as well as target groups and sites. By this time we expect to have a broad user base for the core Web100 software distribution. The central help desk will be primarily responsible for providing support for this broader community. Documentation and training efforts will continue, and we will begin to finalize materials for the core distribution and to develop them for the expanded distribution. By the end of the project we expect to have a completed Web100 software distribution, namely an expanded Web100 software suite coupled with “user friendly” documentation and on-line training materials. This will help promote the adoption of Web100 by a broad user community as well as members of the vendor community.

2.3 Vendor Liaison

As indicated earlier, adoption of all or part of the Web100 software suite by commercial vendors is critical to the long-term success of the project. Such adoption must occur to assure the availability of the Web100 performance improvements to a large user population and a diverse number of operating systems and hardware platforms. During the first year of the project, we will do the groundwork necessary to form connections with the vendor community. During the second two years of the project, we will focus on two specific areas: standardization of the TCP-MIB through the IETF and working with the vendor community to integrate significant portions of the Web100 software suite into their operating systems.

Once the core of the new TCP-MIB has been designed, we will work to develop an Internet draft within the IETF. We will shepherd the TCP-MIB through the IETF process, including developing a reference implementation for the TCP-MIB. We have extensive experience with this process, as indicated by Mr. Mathis’ work on the SACK RFC (RFC2018). Since the IETF standardization process takes time, we will not wait for the RFC to be completed and approved before distributing a Linux version of the TCP-MIB.

We have begun to identify commercial vendors whom we believe to be critical to the wide distribution of the Web 100 suite in the Internet community. During the first year of the project, we will identify and cultivate relationships with specific individuals in each of these companies in order to foster an understanding of our work and its value to their companies. During the second two years of the project, we will broaden these relationships to include the development teams within each company with whom we will work to implement our work in commercial operating systems. Also during the second and third years of Web100, we will work closely with commercial vendors as they implement our algorithms in their commercial products, providing advice to these vendors, as well as receiving feedback.

During the second year of the project, we will work to make distributors aware of the importance of our work in the area of high-performance networking. We will also develop relationships with Linux open source distributors, such as Red Hat, Caldera, Turbo Linux, and others. During the third year of the project, after code development is substantially complete, our goal is to have Web100 changes included in the standard Linux kernel distribution and to have the Web100 tools included in multiple commercially distributed Linux packages.

3. Timeline and Milestones

Over the 3 years of the project, we plan to design, implement and distribute the Web100 software to the scientific user community, providing User Services Support throughout the process. As indicated previously, for Web100 to be truly successful, the Web100 software suite needs to be adopted by the more general user community. The timeline reflects our efforts in this area, specifically in the area of Vendor Liaison.

During Year 1, we plan to focus on the core Web100 technologies: TCP-MIB, FTP and autotuning. Prototypes of all three of these technologies will be designed, developed and distributed to a select group of early adopters. User Support and Vendor Liaison efforts will proceed in parallel with the development work. In User Support, we will begin by surveying the scientific user community to identify their needs and requirements associated with transparent access to the network infrastructure at 100 Mbps data rates. Results from the survey will be used to fine-tune the efforts of the development team as well as to help define the Web100 User Support infrastructure. By the end of Year 1, we will have initial versions of the TCP-MIB, optimized FTP and autotuning available from the Web100 web site. By the end of Year 1 we will have developed and begun a strategy for packaging the Web100 software suite.

During Year 2 and 3 we plan to build on the core technologies and User Support Services infrastructure developed during the first year. We will revise and expand the TCP-MIB. Revisions will be based on feedback from the early adopter community. The TCP-MIB will be expanded to incorporate parameters and features not included in the initial implementation. We will continue the IETF standardization process, helping to author a TCP-MIB Request for Comments (RFC). Implementation of the optimized FTP will be completed by the early part of Year 2. We will provide a production version of autotuning as well as begin development on other applications. Applications currently targeted include diagnostic tools that utilize parameters from the TCP-MIB as well as a graphical user interface that allows user to interactively understand, diagnose and tune their TCP applications.

During Year 2, User Support Services will expand to provide Help Desk services to early adopters as well as the next tier of scientific users. On-site support will be provided to select user groups. In addition to traditional support services, User Services will also provide up-to-date documentation; define and offer Web100 training courses; and provide regular feedback to the development team.

During Year 2, Vendor Liaison work will focus on publicizing the Web100 project and getting support from key vendor groups. By the end of Year 2, we hope to identify a core set of vendors who will adopt all or part of the Web100 software suite. Year 3 will focus on broadening the Web100 user community and completing the Web100 software distribution. Software will be tested, debugged and revised based on feedback from the user community. Production versions of the Web100 software suite will be available in packaged form. New versions are expected to be available every six-months. During this period, the core portions of the Web100 software suite will be ported to a second operating system. Finally, we expect to provide a listing of vendors who support all or part of the Web100 software to the user community.