Dell R650 server
This page contains information out Dell PowerEdge R650 could servers which we have deployed in our cluster.
Documentation and software
Booting and BIOS configuration
Press F2 during start-up to enter the BIOS and firmware setup menus. Go to the BIOS Settings menu.
Minimal configuration of a new server or motherboard
At our site the following minimal settings are required for a new server or a new motherboard.
Remaining settings will be configured by racadm
.
The Dell iDRAC9 (BMC) setup is accessed via the System Setup menu item iDRAC Settings:
In the System Summary page read the NIC iDRAC MAC Address from this page for configuring the DHCP server.
In the Network page set the Enable IPMI over LAN to Enabled.
Go to the System Setup menu item Device Settings and select the Integrated NIC items:
In the NIC Main Configuration Page select NIC Configuration. We use NIC port 3 (1 Gbit) as the system’s NIC.
Read the NIC Ethernet MAC Address from this page for configuring the DHCP server.
Select the Legacy Boot Protocol item PXE.
Performance settings for HPC
The following BIOS settings should be made on HPC nodes:
[Key=BIOS.Setup.1-1#SysProfileSettings]
SysProfile=PerfOptimized
[Key=BIOS.Setup.1-1#SysProfileSettings]
WorkloadProfile=HpcProfile
[Key=BIOS.Setup.1-1#DellControlledTurbo]
ControlledTurbo=Disabled
[Key=BIOS.Setup.1-1#IntegratedDevices]
SnoopHldOff=Roll8KCycles
The CLI commands are:
racadm set BIOS.SysProfileSettings.SysProfile PerfOptimized
racadm set BIOS.SysProfileSettings.WorkloadProfile HpcProfile
racadm set BIOS.DellControlledTurbo.ControlledTurbo Disabled
racadm set BIOS.IntegratedDevices.SnoopHldOff Roll8KCycles
racadm jobqueue create BIOS.Setup.1-1
and then reboot the server.
The SnoopHldOff value (should be 8K or 4K, the default 256 is too small) is important for high performance Omni-Path or Infiniband network fabrics.
System Thermal settings
System Thermal Profile settings can be changed based on the need to maximize performance or power efficiency. This can make CPU thermal throttling less likely.
Read the document Custom Cooling Fan Options for Dell EMC PowerEdge Servers.
In the BIOS setup screen, select iDRAC->Thermal and configure Thermal profile = Maximum performance.
Read the current settings:
racadm get System.ThermalSettings
For HPC applications set the fans to high performance:
racadm set System.ThermalSettings.ThermalProfile "Maximum Performance"
racadm set System.ThermalSettings.MinimumFanSpeed 25
PXE boot setup
Go to the System Setup menu item Device Settings and select the Integrated NIC items:
In the NIC Main Configuration Page select NIC Configuration. We use NIC port 3 (1 Gbit) as the system’s NIC.
Read the NIC Ethernet MAC Address from this page for configuring the DHCP server.
Select the Legacy Boot Protocol item PXE.
Set Wake On LAN to Enabled.
Set the Boot Retry Count = 3 if desired.
Disable PXE boot for all unused NICs (port 1).
Press Finish to save all settings.
iDRAC (BMC) setup
The Dell iDRAC9 (BMC) setup is accessed via the System Setup menu item iDRAC Settings:
In the System Summary page read the NIC iDRAC MAC Address from this page for configuring the DHCP server.
In the Network page set the Enable IPMI over LAN to Enabled.
In the User Configuration page set the User 2 (root) Administrator user name and change the password. The Dell iDRAC default password for root is calvin and you will be asked to change this at the first login.
IMPORTANT: The iDRAC9 keyboard layout is US English! Do not use characters that differ from the US layout!
Optional: In the Thermal page set Thermal: Maximum Performance.
Press Finish to save all settings.