System Configuration
Bridges comprises over 850 computational nodes,
- Regular Shared Memory (RSM) nodes with 128GB RAM each
- Large Shared Memory (LSM) nodes with 3TB RAM each
- Extreme Shared Memory (ESM) nodes with 12TB RAM each
- Regular Shared Memory GPU (RSM-GPU), 48 nodes of two types:
- 16 nodes with 2 Tesla K80 GPUs and 128GB RAM each
- 32 nodes with 2 Tesla P100 GPUs and 128GB RAM each
- AI-GPU (AI-GPU) nodes - ten nodes of two types.
- Nine with 8 Volta V100 GPUs, 16GB GPU memory and 192GB RAM each ("Volta 16")
- An NVIDIA DGX-2, with 16 Volta V100 GPUs, 1.5TB RAM and an NVSwitch, which tightly couples the GPUs for capability and scaling ("Volta 32")
Bridges' computational nodes supply 1.3018 Pf/s and 274 TiB RAM. The Bridges system also includes more than 6PB of node-local storage and 10PB of shared storage in the Pylon file system.
In addtion to its computational nodes, Bridges contains a number of login, database, web server and data transfer nodes.
RSM nodes
| Number | 752 |
| CPUs | 2 Intel Haswell (E5-2695 v3) CPUs; 14 cores/CPU; 2.3 - 3.3 GHz |
| RAM | 128GB, DDR4-2133 |
| Cache | 35MB LLC |
| Node-local storage | 2 HDDs, 4TB each |
| Server | HPE Apollo 2000 |
LSM nodes
| Number | 8 | 34 |
| CPUs | 4 Intel Xeon E5-8860 v3 CPUs; 16 cores per CPU; 2.2 -3.2 GHz | 4 Intel Xeon E7-8870 v4 CPUs; 20 cores/CPU; 2.1 -3.0 GHz |
| RAM | 3TB, DDR4-2133 | 3TB, DDR4-2400 |
| Cache | 40MB LLC | 50MB LLC |
| Node-local storage | 4 HDDs, 4TB each | 4 HDDs, 4TB each |
| Server | HPE ProLiant DL580 | HPE ProLiant DL580 |
ESM nodes
| Number | 2 | 2 |
| CPUs | 16 Intel Xeon E7-8880 v3 CPUs; 18 cores/CPU; 2.3 - 3.1 GHz | 16 Intel Xeon E7-8880 v4 CPUs; 22 cores/CPU; 2.2 - 3.3 GHz |
| RAM | 12TB, DDR4-2133 | 12TB, DDR4-2400 |
| Cache | 45MB LLC | 55MB LLC |
| Node-local storage | 16 HDDs, 4TB each | 16 HDDs, 4TB each |
| Server | HPE Integrity Superdome X | HPE Integrity Superdome X |
RSM-GPU nodes
| Number | 16 | 32 |
| GPUs | 2 NVIDIA Tesla K80 Kepler architecture | 2 NVIDIA Tesla P100 Pascal architecture |
| GPU memory | 12GB/GPU 48GB total/node |
16GB/GPU 32GB total/node |
| CPUs | 2 Intel Haswell (E5-2695 v3) CPUs; 14 cores/CPU; 2.3 - 3.3 GHz | 2 Intel Broadwell E5-2683 v4 CPUs; 16 cores/CPU; 2.1 - 3.0 GHz |
| RAM | 128GB, DDR4-2133 | 128GB, DDR4-2400 |
| Cache | 35MB LLC | 40MB LLC |
| Node-local storage | 2 HDDs, 4TB each | 2 HDDs, 4TB each |
| Server | HPE Apollo 2000 | HPE Apollo 2000 |
| Node names | gpu001 - gpu 016 | gpu017 - gpu048 |
AI-GPU nodes
| Volta 16 | DGX-2 (Volta 32) | |
|---|---|---|
| Number | 9 | 1 |
| GPUs | 8 NVIDIA Volta V100 | 16 NVIDIA Volta V100 |
| GPU memory | 16GB/GPU 128GB total/node |
32GB/GPU 512GB total |
| CPUs | 2 Intel Xeon Gold 6148 20 cores/CPU (40 cores total) 2.4 - 3.7 GHz |
2 Intel Xeon Platinum 8168 24 cores/CPU (48 cores total) 2.7 - 3.7 GHz |
| RAM | 192 GB, DDR4-2666 | 1.5 TB, DDR4-2666 |
| Cache | 33MB | |
| Node-local storage | 4 NVMe SSDs, 2TB each (total 8TB) | 8 NVMe SSDs, 8.84TB each (total ~30TB) |
| Server | HPE Apollo 6500 | - |
| Node names | gpu049 - gpu057 | gpu058 |
Database, web server, data transfer, login nodes
| CPUs | 2 Intel Xeon E5 series CPUs; 14 cores/CPU; 2.3 - 3.3 GHz |
| RAM | 128GB |
| Cache | 35MB LLC |
| Node-local storage | Database nodes have additional SSDs or HDDs |
| Server | HPE ProLiant DL360s or HPE ProLiant DL380s |