HPN-SSH

High performance SSH/SCP

 

 

HPN-SSH is a research project based at the Pittsburgh Supercomputing Center

Theory and Implementation

High Performance Enabled SSH/SCP In Depth

(PI) Chris Rapier PSC, Michael Stevens CMU, Benjamin Bennett PSC, Mike Tasota PSC/CMU, Mitch Dorrell PSC
email: hpn-ssh@psc.edu

Abstract

SCP and the underlying SSH protocol is network performance limited by statically defined internal flow control buffers. These buffers often end up acting as a brake on the network throughput of SCP especially on long and wide paths. Modifying the SSH code to allow the flow control buffers to be defined at run time eliminates this bottleneck.

 

Problem

High Bandwidth and High Latency links are becoming more prevalent in corporate and academic institutions. Applications that use windowing thus need to ensure that the window size is at least equal to the Bandwidth Delay Product, or BDP, are to obtain maximum utilization of the link. The BDP is the product of the narrowest portion of the network path and the round trip delay time and represents the total data carrying capacity of the path. For TCP it is already possible to tune the tcp window size manually or use an autotuning mechanism, such as the Web100 linux kernel patch to ensure maximum throughput with TCP. However, when applications above the TCP layer implement windowing, the limitation on throughput then becomes the less of either TCP or the application. In OpenSSH the limitation appears in the static window sizes that appear in channels.h as defined values.

Solution

Modifying the static size to be a larger value would only serve to waste space in the event that it is larger than the underlying protocol’s window size. Asking the user to specify the size also presents the problem of requiring users to be knowledgable in network performance tuning. Adjusting the size of window to be large enough so that it is no longer the limitation on throughput, but not much larger than it needs to be in order to obtain the desired performance would be the ideal solution.

There were only two changes needed to adjust the SSH window based on the TCP window. One was to enable the buffer code to allocate larger sizes. This was done using a variable that replaced the constant that was the maximum size allowed by the buffer code, and a function to modify the variable’s default value to something larger. The second change was to get the TCP window size from getsockopt and adjust the window size to match, but only if the new size was larger than the old one. The returned value from getsockopt is also doubled because OpenSSH only sends a WINDOW_ADJUST message when the window is half full in order to save on the number of WINDOW_ADJUST messages sent with a cost of doubling the buffer size.

Tests

The following hosts were used in the performance tests. kirana was running a 2.6 linux kernel with the Web100 patch. tg-login was running a 2.6 kernel without autotuning, but a tcp window size of 10,000,000 bytes. The link BDP of a 1Gbps with a 0.04 second delay is 40,000,000 bits or 5,000,000 bytes. The 300MB file was copied from /dev/shm on one machine to /dev/null on the other.

Hosts:

  • kirana.psc.edu
    • Dual PIII 1.0Ghz (Coppermine)
    • 1Gig RAM
    • GigaBit Ethernet
  • tg-login.ncsa.teragrid.org
    • Quad Itanium2 1.3Ghz
    • 8Gig Ram
    • GigaBit Ethernet

Traceroute log:

1 bar-kirana-ge-0-2-0-0.psc.net (192.88.115.169) 0.292 ms 9.452 ms 0.204 ms
2 beast-bar-g4-0-1.psc.net (192.88.115.18) 0.129 ms 0.099 ms 0.094 ms
3 abilene-psc.abilene.ucaid.edu (192.88.115.124) 9.801 ms 9.792 ms 9.805 ms
4 nycmng-washng.abilene.ucaid.edu (198.32.8.84) 14.042 ms 14.036 ms 14.138 ms
5 chinng-nycmng.abilene.ucaid.edu (198.32.8.82) 34.341 ms 41.711 ms 34.326 ms
6 mren-chin-ge.abilene.ucaid.edu (198.32.11.98) 34.421 ms 34.466 ms 34.417 ms
7 sbr0-lsd6509.gw.ncsa.edu (198.17.196.1) 36.957 ms 36.949 ms 36.920ms
8 acb-2-vlan101.gw.ncsa.edu (141.142.0.6) 37.010 ms 36.957 ms 36.943ms
9 core-10-acb-2.gw.ncsa.edu (141.142.0.133) 37.091 ms 36.965 ms 36.958 ms
10 hg-core-core-10.gw.ncsa.edu (141.142.0.138) 38.300 ms 38.866 ms 38.312 ms
11 hg-1-hg-core.ncsa.teragrid.org (141.142.47.34) 38.739 ms 39.187 ms 38.340 ms
12 tg-login1.ncsa.teragrid.org (141.142.48.5) 36.996 ms 36.959 ms 36.950 ms
 

Unmodified SCP Performance

3des-cbc 1.3MB/s
arcfour 1.9MB/s
aes192-cbc 1.8MB/s
aes256-cbc 1.8MB/s
aes128-ctr 1.9MB/s
aes192-ctr 1.8MB/s
aes256-ctr 1.8MB/s
blowfish-cbc 1.9MB/s
cast128-cbc 1.7MB/s
rijndael-cbc@lysator.liu.se 1.8MB/s
 

Modified SCP Performance

Graph of Modified SCP

3des-cbc 2.8MB/s
arcfour 24.4MB/s
aes192-cbc 13.3MB/s
aes256-cbc 11.7MB/s
aes128-ctr 12.7MB/s
aes192-ctr 11.7MB/s
aes256-ctr 11.3MB/s
blowfish-cbc 16.3MB/s
cast128-cbc 7.9MB/s
rijndael-cbc@lysator.liu.se 12.2MB/s
 

Analysis

The tests showed that throughput was increased dramatically, and the limitation was no longer the TCP or SSH window size, but the ability of the host to encrypt at a rate fast enough to send out over the Gigabit Ethernet. This is clearly demonstrated by the vast performance difference between 3des-cbc, the slowest cipher, and arcfour, the fastest cipher.

Security implications

There are no implications that we know of with the following caveats: The use of the None cipher will transfer bulk data in the clean and you must use it at your own risk. The authentication process, however, is fully encrypted. This should, naturally, be seen as riskier than transferring data with full encryption. None encryption is only available in non-interactive sessions. This means scp, batch sftp, and pipes.

Likewise, disabling the use of the Message Authentication Cipher (MAC) has security implications. Specifically, that the data is no longer protected against Man In the Middle (MITM) attacks. As with the None cipher this mode should not be used with sensitive data. The user will need to determine how to properly balance risk versus performance. Disabling the MAC is only available when used with the None ciphers.

Both the client and server must support the None cipher and disabling the MAC for these options to work.

Notes and News

HPN-SSH 18.4.2 Released

HPN-SSH 18.4.2 has been released and is now available on https://github.com/rapier1/hpn-ssh. This release patches the CVE-2024-6387 (aka Regresshion) security flaw. Fedora and Debian packages are also available.”

HPN-SSH 18.4.0 Released

We are pleased to announce that HPN-SSH 18.4.0 has been released and is now available on https://github.com/rapier1/hpn-ssh. This release incorporates OpenSSH 9.7p1. Fedora and Debian packages are also available.

HPN-SSH 18.3.0 Released

We are pleased to announce that HPN-SSH 18.3.0 has been released and is now available on https://github.com/rapier1/hpn-ssh. This release incorporates OpenSSH 9.6p1. Fedora and Debian packages are also available.

HPN-SSH 18.2.0 Released

We are pleased to announce that HPN-SSH 18.2.0 has been released and is now available on https://github.com/rapier1/hpn-ssh. This release incorporates OpenSSH 9.5p1. Fedora and Debian packages are also available.

HPN-SSH 18.1.0 Released

We are pleased to announce that HPN-SSH 18.1.0 has been released and is now available on https://github.com/rapier1/hpn-ssh. This version introduces a parallel ChaCha20-Poly1305 cipher that is 59% faster than OpenSSH 9.4. Packages for Ubuntu and Fedora will be...

Binary Packages Released for 17v11

Binary packages for Debian and Fedora are now available. Commands to add these package sources are:For Debian: sudo add-apt-repository ppa:rapier1/hpnsshFor Fedora: sudo dnf copr enable rapier1/hpnssh

HPN-SSH 17v11 Released

We are pleased to announce that HPN-SSH 17v11 has been released and is now available on https://github.com/rapier1/openssh-portable. This version brings HPN-SSH up to parity with OpenSSH 9.2 and resolves a number of minor bugs. Patches and binary packages will be...

HPN17v0 Released

We are pleased to announce HPN17v0. This version brings us up to date with OpenSSH 8.9. HPN17 also brings a big change to how we are naming things. From now all executables will have an "hpn" prefix attached to them. So "ssh" is now "hpnssh" and "scp" becomes...

Patch sets on SourceForge

All patch sets from 4.4p1 to 8.1p1 are now available on SourceForge at https://sourceforge.net/projects/hpnssh/. The entire codebase (merged with OpenSSH) is also available as a git repo from https://github.com/rapier1/openssh-portable. The SourceForge location now...

NSF funds HPN-SSH

We are proud to announce that the HPN-SSH development team has received a grant from the National Science Foundation (Award#: 2004012) to continue development on HPN-SSH. This grant will be used to develop and incorporate new features and optimizations. This grant...


This work was made possible in part by grants from Cisco Systems, Inc., the National Science Foundation, and the National Library of Medicine.