HP DL160 G6 Intel Nehalem compute node 

In May/June 2009 we expect to receive the first HP DL160 G6 Intel Nehalem compute nodes.

HP product info and support 

Product information and specifications for HP ProLiant DL160 G6 Server series. There are Quickspecs and Manuals for DL160 G6.

There is a page with DL160 G6 drivers and downloads for Red Hat Enterprise Linux 5 Server (x86-64).

DL160 CPLD upgrade 

False error messages about power supplies will be logged to the system. This problem does not affect the operation of the server, but is still annoying.

The Advisory c02499696 ProLiant 100-Series Servers - Certain ProLiant 100-Series Servers Configured with HP ProLiant Server Health Driver Version 8.50 (or Earlier) And Red Hat Linux 5 Update 4 May Display False Power Supply Status Messages states:

Certain ProLiant 100-series servers configured with the HP ProLiant Server Health Driver Version 8.50 (or earlier) and Red Hat Linux 5 Update 4 may, on rare occasions, display the following false power supply status messages in the /var/log/messages file:
Jun 5 01:08:09 localhost hpasmpld[4181]: CRITICAL: System Power Supply Removed (Power Supply 1)
Jun 5 01:08:19 localhost hpasmpld[4181]: NOTICE: System Power Supply Inserted (Power Supply 1)
The error message may be logged every few days or weekly. This typically occurs when a single power supply is installed in the server.

This also applies to Windows servers, see Advisory c03027164.

The solution is to download and install SoftPaq SP50423 CPLD Version 1.12 from https://downloads.hpe.com/pub/softlib2/software1/pubsw-windows/p3231552/v64980/SP50423.exe

Installation instructions are missing, but they are:

Execute SP50423.exe on a Windows PC.
Go to the folder C:\\swsetup\\SP50423 and create either a bootable CD or a bootable USB key from the subfolders.
Boot the DL160 G6 server to the CD or USB key and follow instructions.
After the CPLD version 1.12 upgrade is completed, physically unplug power for 1 minute.

DL160 G6 Diagnostics 

To perform a hardware inventory or hardware diagnostics, you need to boot the HP ProLiant DL160 G6 Easy Set-up CD downloadable from the above mentioned downloads page. With this CD there is a tool to create a bootable USB memory stick, however, to enable boot from USB devices see the advice below.

Select the HP Insight Diagnostics Offline Edition for inventory or diagnostics.

Please note that version 2.00 has a bug described in this Advisory: Processor Temperature Test” Incorrectly Fails Under HP Insight Diagnostics Offline Edition 8.2.5.3157 When Launched from the Easy Setup CD. For each processor, uncheck the System->Processor Package 1->Processor Temperature Test to prevent the test from running and therefore returning an incorrect result.

DL160 G6 BIOS settings 

A number of factory default BIOS settings must be changed:

Advanced->CPU Configuration:
*Intel VT-d*                      = Disabled (Virtualization I/O - was in Advanced menu until BIOS 20100519).
*Intel Virtualization Technology* = Disabled (Virtualization).
*Intel HT technology*             = Disabled (Hyperthreading).
The Advanced->SATA Configuration: = Compatibility MUST be changed prior to OS installation:
SATA#1 Controller Mode = AHCI
Advanced: Power Efficiency Mode = Performance.

The Boot->Standard Boot Order: is changed so that Network is before Hard drive.

The Boot->Boot Settings Configuration: is changed so that Num Lock = Disabled.

The Boot->Boot Settings Configuration: is changed so that Quiet Boot = Disabled.

Booting from an USB device 

The HP DL100 series servers’ BIOS seem to detect USB disk devices and memory sticks as a built-in hard disk. This means that the boot order is confusing, but you can boot different devices like this:

USB CD-ROM drives will always be the 1st boot device (maybe there’s a BIOS setting to modify this?).
USB disk devices and memory sticks will become hard drive #1, making the built-in hard disk #2.
If PXE is the first boot device, PXE booting can be interrupted by pressing the keyboard ESC button, causing the BIOS to boot from the hard disk #1 device, which is the USB disk device (if inserted before starting the server).

If you need to boot a HP Diagnostics or other USB disk device, use method 3. above.

CentOS installation on DL160 G6 

Disk performance problem 

If you use the:

SATA Configuration = Compatibility

BIOS setting, you will have serious disk performance problems. This can to some extent be alleviated by the procedure below, but it is recommended to use the AHCI setting as shown above.

The initial CentOS 5.3 installation has very bad disk performance even with pci=conf1, and this needs to be replaced in /etc/grub.conf by:

pci=nomsi,nommconf hda=noprobe hdc=noprobe

See https://bugzilla.redhat.com/show_bug.cgi?id=439391 (postings at the end).

Note: after the kickstart installation please make sure that /etc/grub.conf contains the necessary kernel flags.

CentOS 5.3 does not fix this problem (tested with kernel 2.6.18-128.1.6.el5PAE), so noprobe flags are still needed. This solution obsoletes the kernel replacement stuff documented below.

DL160 G6 BIOS upgrades 

BIOS upgrade from Linux 

BIOS upgrades can be performed from Linux using the Online ROM Flash Component for Linux - HP ProLiant DL160 G6 Servers from the above download page.

Run the script as root, for example:

sh CP010987.scexe

BIOS upgrade from PXE boot 

HP also delivers ROMPAQ BIOS upgrades for USB key media, and flat files, in a Windows executable .EXE file. However, the BIOS flat files can be copied to a 1.44 MB diskette image which may be used as a PXE-bootable image file.

Unpack the ROMPAQ SPxxxxx.EXE file on a Windows PC and copy the folders to a Linux system.

Before running an unattended BIOS upgrade from DOS, first check the available command flags which are:

> rompaq.exe /?

Usage:    ROMPAQ [options]
Options:
 /B[ootblock]   (Simulate disaster recovery mode)
 /C[ompress]    infile product_id rom_index rom_revision
                [iterate_size (in Kbytes)]
                (Compress infile using specified args in header)
 /CS            infile
                (Short form compression.  Driver must support this feature.
 /D[ecompress]  infile outfile
                (Decompress infile to outfile)
 /F[actory]     driver product_id
                (Run in factory mode and update specified product ROM)
 /H[elp] or /?  (Display this message)
 /R[emote]      (Run in remote mode)
 /S[pawned]     (Return non-zero val if spawning program should reboot)
 /T[ext]        (Use text mode interface instead of graphics)
 /U[nattended]  driver image_file
                (Run in unattended mode and update specified product ROM)
 /V[erify]      infile
                (Verify the checksum of infile)
 /X [test]      (Pass test mode flag to drivers)
 /! [Halt]      (Halt machine instead of exiting to DOS)

Creating a bootable DOS diskette image 

We need to obtain a DOS bootable diskette image from some alternative source. A very minimal DOS image is required because of the 1.44 MB diskette image size limit.

We have used the http://www.bootdisk.com/bootdisk.htm image Special Disk For Bios Flashing. Unzip the DRDFLASH files to an empty directory and mount the image:

mkdir drdflash
cd drdflash
unzip ../drdflash.zip
cp DRDFLASH.IMG /tmp/dl160g6_bios.img
mount -o loop /tmp/dl160g6_bios.img /mnt

Changing boot diskette from 1.44MB to 2.88MB 

The files to be copied to the boot diskette image may cause a disk full error on 1.44MB diskettes.

It is however possible to change the DRDFLASH.IMG image from 1.44MB to 2.88MB using the shareware tool WinImage on a Windows PC. Open the DRDFLASH.IMG image with WinImage and select the menu item Image->Change format… and select Standard format: 2.88 MB. Save the image and exit WinImage.

Creating a ROMPAQ boot diskette image 

Mount the above mentioned DOS bootable diskette image as described above.

First copy the ROMPAQ files to a Linux PC as described above. Then copy the ROMPAQ files to the image:

cp .../ROMPaq Flat Files/* /mnt/

Edit the DRDFLASH file /mnt/autoexec.bat (make sure that the file contains CR-LF at the end, use od -c autoexec.bat) to append a line similar to:

ROMPAQ /T /U CPQSRVR2.CPU CPQO3302.xxx

Here CPQSRVR2.CPU is the driver and CPQO3302.xxx is the image file. Remember to change the filename CPQO3302.xxx in autoexec.bat according to the filenames in the ROMPAQ !

The /U and /T flags should be used for unattended BIOS upgrades.

Unmount the image and copy it to /tftpboot on the TFTP boot server:

umount /mnt
scp /tmp/dl160g6_bios.img root@ymer:/tftpboot/dl160g6/

For standard interactive BIOS upgrade the autoexec.bat file should contain:

ROMPAQ.EXE /L:US /!

This line can only be seen from the USB key upgrade media, it doesn’t seem to be documented (and the /L flag is undocumented).

AFUDOS BIOS upgrades 

BIOS upgrades newer than 07/06/2011 use the AMI BIOS AFUDOS.EXE AMI FLASH UTILITY for DOS. The default AUTOEXEC.BAT file on the DOS upgrade image calls a file FBB.BAT containing:

echo Flashing Main BIOS & Boot Block...
afudos 18DIM243.ROM /p /b /s /q

This will do an unattended BIOS upgrade and leave at the DOS prompt, from which the user must reboot manually. To list all command flags do:

afudos /h

DL160 G6 BIOS parameter replication 

The DL160 G6 BIOS parameter settings can be saved or restored by means of the HP SmartStart Scripting Toolkit for Linux (see the downloads page). However, version 2.20 or greater of this toolkit is required for use with the DL100 series servers.

When running under Linux (using e.g. CentOS or SmartStart), use this command on a DL160 G6 node to save the BIOS settings to a file:

cd /home/ymer/SmartStartScripting-2.20/utilities
./conrep -s -f /tmp/conrep.xml -x ../conrep_xml/ConrepDL160G6.xml

The file /tmp/conrep.xml contains the BIOS settings that can be restored by:

  cd /home/ymer/SmartStartScripting-2.20/utilities
./conrep -l -f /tmp/conrep.xml -x ../conrep_xml/ConrepDL160G6.xml

The conrep command syntax is:

Usage conrep -s | -l [-f output filename] [-x xml configuration filename] [-?]

-s Saves the current configuration to a file.
-l Loads configuration setting from a file.
-f Name of the output file.
-x Name of the XML definition file.
      If not present, the XML configuration will default to conrep.xml
      If not present, the output filename default to conrep.dat

Error Codes:
      0 - Success
      1 - Bad XML File
      2 - Bad Data File
      4 - Admin Password set
      5 - No XML Tag

DL160G6 BMC 

The on-board BMC controller is documented in the manual Lights-Out 100 User Guide.

The BMC MAC address is found in the Setup screen IPMI Configuration->Lan Configuration where you can also configure DHCP. You enable the LO100 Shared Ethernet network access as described on pages 7 and 9 with the IP-address being assigned by DHCP:

Boot into BIOS Setup (F10).
Go to the setup window IPMI Configuration->Set LAN Configuration.
Set the item BMC NIC Allocation to Shared (default is Dedicated).

You can also use the IPMI interface to configure the BMC, but you cannot change the BMC NIC Allocation!

BMC firmware updates 

HP has released BMC firmware updates that can be executed under Linux. In the software download page see the item Online Firmware Flash Component for Linux - HP ProLiant G6/G7 Lights-Out 100 Remote Management which was first released in June 2010.

Download and execute the BMC firmware update file on-line under Linux. However, due to the bad way HP’s utilities handle the IPMI interface, special care must be done when upgrading the BMC. You must manually handle the starting and stopping of IPMI in the following way:

service ipmi restart  # Initialize IPMI correctly
sh CP013308.scexe     # Run firmware upgrade tool
service ipmi restart  # Reinitialize IPMI correctly after upgrade
service ipmi stop     # Stop IPMI correctly

If you neglect to perform the mentioned IPMI commands, a superfluous CPU load of 1.0 will be permanently added to your system!

BMC firmware updates on CentOS6/RHEL6 

On CentOS6/RHEL6 the BMC firmware update script doesn’t work, you get this error:

# ./CP017117.scexe
./hpsetup: line 2: .: lo100.sh: file not found
Parameters are: --source /root
./hpsetup: line 7: lo100main: command not found

First you must install this package:

yum install libstdc++-4.4.6-3.el6.i686

The problem seems to be the newer version of bash in RHEL6. A simple workaround is to prepend “.” to the PATH:

PATH=.:${PATH}
./CP017117.scexe

See advice in http://jfut.integ.jp/2012/04/20/update-hp-firmware-error-on-rhel6-another/.

Another method: Extract the contents of the CPxxxxxx.scexe package:

./CP017117.scexe  --unpack=setup

which will extract to the directory setup. In the script hpsetup change the shell from sh to bash, see http://jfut.integ.jp/2012/03/18/update-hp-firmware-error-on-rhel6/. Then run the hpsetup script.

Alternative method of network updating 

The section “Remotely updating the firmware” on the manual’s page 11 first creates an uncompressed binary image file:

rompaq /d CPQQ1402.272 ldrimage.bin

You can now telnet to the BMC IP-address using the default username/password of admin/admin.

To upgrade the BMC firmware from the telnet prompt:

cd map1/firmware
load -source //10.2.128.2/dl160g6/LDRIMAGE.BIN -oemhpfiletype csr

However, this doesn’t seem to work (June 25, 2009), and neither does update from a web browser.

DL100 series SETSYS (setting the serial number)

When the system board is replaced, the Setsys Utility for HP ProLiant 100 series G5 and G6 Servers (For USB Key-Media) must be booted to set the serial number in the new system board.

You can find updated versions of SETSYS under the DL160 G5 with DR-DOS operating system (not yet with DL160 G6).

To make a PXE-bootable image with SETSYS, prepare a minimal DOS image as shown above for the BIOS. Unpack the HP utility and copy the files from the Flat Files folder to the image.

Usage information for SETSYS is obtained by:

SETSYS /?

The HP SETSYS utility command to update the serial number must be typed in UPPER CASE:

Type the following (with the appropriate Serial Number):
```
SETSYS /S=GBxxxxxxx
```
Run just SETSYS (no switches) to verify change.

DL160 power supply 

Our DL160 G6 nodes all use the HP High Efficiency 460W Power Supply Kit (P/N 535684-B21). If you want to buy a spare part power supply, the part number is P/N 511777-001.