Dell R650 server

This page contains information out Dell PowerEdge R650 could servers which we have deployed in our cluster.

Documentation and software

Dell Support provides R650 information:

Booting and BIOS configuration

Press F2 during start-up to enter the BIOS and firmware setup menus. Go to the BIOS Settings menu.

Minimal configuration of a new server or motherboard

At our site the following minimal settings are required for a new server or a new motherboard. Remaining settings will be configured by racadm.

The Dell iDRAC9 (BMC) setup is accessed via the System Setup menu item iDRAC Settings:

  • In the System Summary page read the NIC iDRAC MAC Address from this page for configuring the DHCP server.

  • In the Network page set the Enable IPMI over LAN to Enabled.

Go to the System Setup menu item Device Settings and select the Integrated NIC items:

  • In the NIC Main Configuration Page select NIC Configuration. We use NIC port 3 (1 Gbit) as the system’s NIC.

  • Read the NIC Ethernet MAC Address from this page for configuring the DHCP server.

  • Select the Legacy Boot Protocol item PXE.

Boot settings menu

  • The Boot Mode may be either BIOS or UEFI.

  • In the Boot Sequence menu:

    • Click the Boot Sequence item to move PXE boot up above the hard disk boot (if desired).

    • Verify that the correct devices are selected in Boot Option Enable/Disable.

If UEFI boot mode is selected, the following must be enabled before installing the OS for the first time:

  • In the Boot Setting menu:

    • Hard-disk Drive Placeholder = Enabled

Performance settings for HPC

The following BIOS settings should be made on HPC nodes:

[Key=BIOS.Setup.1-1#SysProfileSettings]
SysProfile=PerfOptimized
[Key=BIOS.Setup.1-1#SysProfileSettings]
WorkloadProfile=HpcProfile
[Key=BIOS.Setup.1-1#DellControlledTurbo]
ControlledTurbo=Disabled
[Key=BIOS.Setup.1-1#IntegratedDevices]
SnoopHldOff=Roll8KCycles

The CLI commands are:

racadm set BIOS.SysProfileSettings.SysProfile PerfOptimized
racadm set BIOS.SysProfileSettings.WorkloadProfile HpcProfile
racadm set BIOS.DellControlledTurbo.ControlledTurbo Disabled
racadm set BIOS.IntegratedDevices.SnoopHldOff Roll8KCycles
racadm jobqueue create BIOS.Setup.1-1

and then reboot the server.

The SnoopHldOff value (should be 8K or 4K, the default 256 is too small) is important for high performance Omni-Path or Infiniband network fabrics.

System Thermal settings

System Thermal Profile settings can be changed based on the need to maximize performance or power efficiency. This can make CPU thermal throttling less likely.

Read the document Custom Cooling Fan Options for Dell EMC PowerEdge Servers.

In the BIOS setup screen, select iDRAC->Thermal and configure Thermal profile = Maximum performance.

Read the current settings:

racadm get System.ThermalSettings

For HPC applications set the fans to high performance:

racadm set System.ThermalSettings.ThermalProfile "Maximum Performance"
racadm set System.ThermalSettings.MinimumFanSpeed 25

Processor settings menu

You may want to verify the following BIOS settings on HPC nodes:

  • Disable Hyperthreading by Logical Processor = Disabled.

  • Virtualization Technology = Disabled.

  • Dell Controlled Turbo = Disabled.

  • Sub NUMA Cluster = Enabled.

The Sub NUMA Cluster (SNC, replaces the older Cluster-on-Die (COD) implementation) has been shown to improve performance, see BIOS characterization for HPC with Intel Cascade Lake processors. This will cause each processor socket to have two NUMA domains for the two memory controllers, so a dual-socket server will have 4 NUMA domains.

Display the NUMA domains by:

$  numactl --hardware
available: 4 nodes (0-3)
...

System Profile Settings menu

  • System Profile = Performance.

System Security menu

  • AC Power Recovery = Last state.

Miscellaneous Settings menu

  • Keyboard NumLock = Off.

PXE boot setup

Go to the System Setup menu item Device Settings and select the Integrated NIC items:

  • In the NIC Main Configuration Page select NIC Configuration. We use NIC port 3 (1 Gbit) as the system’s NIC.

  • Read the NIC Ethernet MAC Address from this page for configuring the DHCP server.

  • Select the Legacy Boot Protocol item PXE.

  • Set Wake On LAN to Enabled.

  • Set the Boot Retry Count = 3 if desired.

  • Disable PXE boot for all unused NICs (port 1).

Press Finish to save all settings.

iDRAC (BMC) setup

The Dell iDRAC9 (BMC) setup is accessed via the System Setup menu item iDRAC Settings:

  • In the System Summary page read the NIC iDRAC MAC Address from this page for configuring the DHCP server.

  • In the Network page set the Enable IPMI over LAN to Enabled.

  • In the User Configuration page set the User 2 (root) Administrator user name and change the password. The Dell iDRAC default password for root is calvin and you will be asked to change this at the first login.

    IMPORTANT: The iDRAC9 keyboard layout is US English! Do not use characters that differ from the US layout!

  • Optional: In the Thermal page set Thermal: Maximum Performance.

Press Finish to save all settings.