Hardware information
Niflheim overview
Niflheim currently (as of February 2024) consists of the following hardware:
A total of 700 compute nodes.
The nodes contain a total of 31000 CPU cores.
Aggregate theoretical peak CPU performance of more than 2255 TeraFLOPS (TFLOPS) or 2.2 PetaFLOPS.
Total aggregate RAM memory is 308 TB.
CPU and GPU architectures
Niflheim comprises several generations of hardware for a total of 700 compute nodes:
Lenovo AMD EPYC 9474F servers
68 96-core nodes Lenovo SD665_V3 with two AMD EPYC Zen4 9474F 48-core CPUs (a total of 6528 cores) running at 3.60 GHz Base frequency (up to 4.10 GHz in Boost Clock) and including AVX-512 vector instructions. The Direct Water Cooled servers are mounted in Lenovo Neptune DW612S chasisses.
Peak speed: 6500 GFLOPS/node, 442 TFLOPS in total for all nodes.
The RAM memory is 768 GB (8.0 GB/core) 4800 MHz DDR5 dual-rank memory, 52.2 TB in total for all nodes.
Each server has a 1.9 TB local NVMe hard disk.
The network interconnect is 100 Gbit/s water-cooled ThinkSystem NVIDIA ConnectX-7 NDR200 InfiniBand QSFP112 Adapters.
Installed in February 2024.
Lenovo Intel servers with 4 TB RAM memory
4 32-core nodes Lenovo SR850_V3 2U servers. The four processors are Intel(R) Xeon(R) Gold 6434H running at 3.70 GHz Base frequency (up to 4.10 GHz in Turbo_mode) and including AVX-512 vector instructions..
The RAM memory is 4096 GB (128 GB/core) 4800 MHz DDR5 dual-rank memory, 16 TB in total for all nodes.
Peak speed: 3789 GFLOPS/node, 15 TFLOPS in total for all nodes.
Each server has 15 TB local NVMe scratch disk (striped over 4 disks of 3.9 TB for performance).
Installed in February 2024.
Lenovo GPU servers with 4xA100 SXM4
4 GPU nodes Lenovo SD650-N_V2 1U servers that feature the Intel Xeon family of processors and 4 times NVIDIA A100 A100-SXM4-40GB GPUs and an NVLink interconnect.
The processors are Intel Xeon Platinum 8358 2 times 32-core CPUs running at 2.60 GHz Base frequency (up to 3.40 GHz in Turbo_mode) and including AVX-512 vector instructions.
The RAM memory is 512 GB 3200 MHz DDR4 dual-rank memory per node, 2 TB in total for all nodes.
CPU peak speed: 5325 GFLOPS/node, 21 TFLOPS in total for all nodes.
The Direct Water Cooled servers are mounted in Lenovo Neptune DW612S chasisses.
Installed in February 2024.
Dell AMD server with Instinct MI210 GPU
Dell Ice Lake servers
96 56-core nodes Dell R650 with two Intel IceLake Xeon_Gold_6348 28-core CPUs (a total of 5376 cores) running at 2.60 GHz Base frequency (up to 3.50 GHz in Turbo_mode) and including AVX-512 vector instructions.
Peak speed: 4659 GFLOPS/node, 447 TFLOPS in total for all nodes.
The RAM memory is 512 GB (9.14 GB/core) 3200 MHz DDR4 dual-rank memory, 49.1 TB in total for all nodes.
Each server has a 480 GB local SSD hard disk.
The network interconnect is 100 Gbit/s Cornelis_Networks (previously Intel) OmniPath.
Installed in December 2021.
SuperMicro servers
8 40-core nodes SYS-4029GP-TRT2 from Nextron/SuperMicro with Intel Xeon_Gold_5218R 20-core CPUs @2.10 GHz (a total of 320 cores). Hyperthreading is enabled so that each node has 80 threads or virtual CPUs.
Peak speed: 2688 GFLOPS/node, 21 TFLOPS in total for all nodes.
The RAM memory is 198 GB (4 nodes) and 768 GB (4 nodes), 3.81 TB in total for all nodes.
These servers are equipped with Nvidia GPUs.
Each server has a 960 GB local NVMe hard disk and a 1 Gbit Ethernet.
Installed in December 2020 and July 2021.
Dell Cascade Lake Refresh servers
128 40-core nodes Dell R640 with two Intel Cascade_Lake Xeon_Gold_6242R 20-core CPUs (a total of 5120 cores) running at 3.10 GHz Base frequency (up to 4.10 GHz in Turbo_mode) and including AVX-512 vector instructions.
Peak speed: 3968 GFLOPS/node, 508 TFLOPS in total for all nodes.
The RAM memory is 384 GB (9.6 GB/core) 2933 MHz DDR4 dual-rank memory, 49.1 TB in total for all nodes.
Each server has a 240 GB local SSD hard disk.
The network interconnect is 100 Gbit/s Cornelis_Networks (previously Intel) OmniPath.
Installed in October 2020.
Dell Skylake servers
208 40-core nodes Dell C6420 and R640 with two Intel Skylake Xeon_Gold_6148 20-core CPUs (a total of 8320 cores) running at 2.40 GHz Base frequency (up to 3.70 GHz in Turbo_mode) and including AVX-512 vector instructions.
Peak speed: 3072 GFLOPS/node, 639 TFLOPS in total for all nodes.
The RAM memory type is 2666 MHz DDR4 dual-rank memory:
196 C6420 nodes have 384 GB of memory (9.6 GB/core), 75.3 TB in total for all nodes.
12 R640 nodes have 768 GB of memory (19.2 GB/core), 9.2 TB in total for all nodes.
Each server has a 240 GB local SSD hard disk.
The network interconnect is 100 Gbit/s Cornelis_Networks (previously Intel) OmniPath.
Installed in April 2019.
Huawei Broadwell servers
192 24-core nodes Huawei XH620 v3 with two Intel Broadwell Xeon_E5-2650_v4 12-core CPUs (a total of 4608 cores) running at 2.20 GHz (up to 2.90 GHz in Turbo_mode).
Peak speed: 845 GFLOPS/node, 162 TFLOPS in total for all nodes.
The RAM memory type is 2400 MHz DDR4 dual-rank memory:
180 nodes have 256 GB of memory (10.7 GB/core), 46.1 TB in total for all nodes.
12 nodes have 512 GB of memory (21.3 GB/core), 6.1 TB in total for all nodes.
Each server has a 240 GB local SSD hard disk.
The network interconnect is 100 Gbit/s Cornelis_Networks (previously Intel) OmniPath.
Installed in December 2016, March 2017, November 2017.
File servers
Several Linux file servers are available for the departmental user groups. Each group is assigned a file-system on one of the existing file servers. Depending on disk requirements, group file-systems can be from 1 TB and up.
The file servers are standard Linux servers with large disk arrays, sharing the file-systems using NFS. We do not use any parallel file servers (for example, Lustre etc.).
The file server total available disk spaces are:
Server niflfs1: 108 TB
Server niflfs3: 87 TB
Server niflfs4: 90 TB
Server niflfs5: 90 TB
Server niflfs6: 106 TB
Server niflfs7: 106 TB
Server niflfs8: 163 TB
Server niflfs9: 163 TB
A maximum disk capacity of 913 TB disk space is available for user applications.