Bridges-2
Bridges-2, PSC’s flagship supercomputer, began production operations in March 2021. It is funded by a $10-million grant from the National Science Foundation.
Bridges-2 provides transformative capability for rapidly evolving computation- and data-intensive research, and creates opportunities for collaboration and convergence research. It supports both traditional and non-traditional research communities and applications. Bridges-2 integrates new technologies for converged, scalable HPC, machine learning and data; prioritizes researcher productivity and ease of use; and provides an extensible architecture for interoperation with complementary data-intensive projects, campus resources, and clouds.
Bridges-2 is available at no cost for research and education, and at cost-recovery rates for other purposes.
Core concepts and innovation
Bridges-2 was designed around the core concepts of converged HPC + AI + Data; a custom topology optimized for data-centric HPC, AI, and HPDA; heterogeneous node types for different aspects of workflows, including both CPUs and AI-targeted GPUS; 3 tiers of per-node RAM: 256GB, 512GB, and 4TB; an extremely flexible software environment; and community data collections and Big Data as a Service.
Innovative features built into Bridges-2 to facilitate these core concepts include:
- AMD EPYC 7742 CPUs: 64-core2.25–3.4 GHz
- AI scaling to 192 V100-32GB SXM2 GPUs
- 100TB, 9M IOPs flash array accelerates deep learning training, genomics, and other applications
- Mellanox HDR-200 InfiniBand doubles bandwidth & supports in-network MPI-Direct, RDMA, GPUDirect, SR-IOV, and data encryption
- Cray ClusterStor E1000 Storage System
- HPE DMF single namespace for data security and expandable archiving
Questions?
If you have questions about Bridges-2, please email help@psc.edu.
Node types
Regular Memory nodes
Regular Memory (RM) nodes provide extremely powerful, general-purpose computing, machine learning and data analytics, AI inferencing, and pre- and post-processing.
488 RM nodes have 256GB of RAM, and 16 have 512GB of RAM.
Details
- NVMe SSD (3.84TB)
- Mellanox ConnectX-6 HDR Infiniband 200Gb/s Adapter
- Two AMD EPYC 7742 CPUS, each with:
- 64 cores
- 2.25-3.40GHz
- 256MB L3
- 8 memory channels
Extreme Memory nodes
Extreme Memory (EM) nodes provide 4TB of shared memory for genome sequence assembly, graph analytics, statistics, and other applications requiring a large amount of memory for which distributed-memory implementations are not available.
Details
Each of Bridges-2’s 4 EM nodes consists of:
- 35.75MB LLC 6 memory channels
- 4TB of RAM: DDR4-2933
- NVMe SSD (7.68TB)
- Mellanox ConnectX-6 HDR InfiniBand 200Gb/s Adapter
- Four Intel Xeon Platinum 8260M “Cascade Lake” CPUs:
- 24 cores
- 2.40–3.90GHz
GPU nodes
GPU nodes provide exceptional performance and scalability for deep learning and accelerated computing. Bridges-2 launched with 24 GPU nodes, totaling 40,960 CUDA cores and 5,120 tensor cores. The GPU nodes from Bridges were later migrated to Bridges-2, adding the DGX-2 and nine more V100 GPU nodes to Bridges-2’s GPU resources.
Details
The 24 original Bridges-2 GPU nodes contain:
- 512GB of RAM: DDR4-2933
- 7.68TB NVMe SSD
- Two Mellanox ConnectX-6 HDR Infiniband 200Gb/s Adapter
- Eight NVIDIA Tesla V100-32GB SXM2 GPUs
- 1 Pf/s tensor
- Two Intel Xeon Gold 6248 “Cascade Lake” CPUs:
- 20 cores, 2.50–3.90GHz, 27.5MB LLC, 6 memory channels
The Bridges GPUs migrated to Bridges-2 include nine nodes with:
- 192GB of RAM, DDR4-2666
- 4 NVMe SSDs, 2TB each
- 8 NVIDIA V10016GB GPUs
- 2 Intel Xeon Gold 6148 CPUs:
- 20 cores/CPU, 2.4 – 3.7 Ghz
The DGX-2 has:
- 1.5TB RAM, DDR4-2666
- 8 NVMe SSDs, 8.84TB each
- 16 NVIDIA Volta V100-32GB GPUs
- 2 Intel Xeon Platinum 8168 CPUs:
- 24 cores/CPU, 2.7 – 3.7 GHz