Install EL Linux via PXE and UEFI

Overview

This HowTo guide documents how to install EL/RHEL Linux using PXE on a client host booting by UEFI. We will show how to support UEFI booting with PXE, downloading files from your TFTP server. See also our pages:

Other useful pages:

UEFI security

There is an NSA Security Report UEFI DEFENSIVE PRACTICES GUIDANCE with some recommendations:

  • Machines running legacy BIOS or UEFI in compatibility mode should be migrated to UEFI native mode to take advantage of new features.

  • UEFI should be secured using a set of administrator and user passwords appropriate for a device’s capabilities and intended use.

  • Firmware comprising UEFI should be updated regularly and treated as importantly as Operating System (OS) updates.

  • UEFI Secure Boot should be enabled and configured to audit firmware modules, expansion devices, and bootable OS images.

  • Trusted Platform Module (TPM) should be leveraged to check the integrity of UEFI.

See also:

UEFI network boot process

In this section we describe how a computer doing an UEFI network PXE boot will download a bootloader image file from the network’s TFTP server and execute it. For 64-bit UEFI systems with the x86-64 architecture, the boot file name by convention is BOOTX64.EFI, but other bootloader images such as shimx64.efi may be used in stead. The bootloader image file is located in the folder /boot/efi/ on a bootable drive. Other CPU architectures than x86-64 are listed in the UEFI_specification section 3.5.

The Linux boot process is explained in detail in Guide to the Boot Process of a Linux System and Booting process of Linux. When powering up the client computer, PXE network booting can be selected using the console, typically by pressing the F12 or F10 Function_key.

When you Enable UEFI support in the DHCP server, and subsequently network boot the client computer, it will first download the bootloader image. This image is executed in the client computer’s UEFI capable NIC adapter, and it will subsequently download the main bootloader image grubx64.efi from the TFTP server, which loads the installation Linux_kernel and initrd (see the Kernel section in Booting process of Linux).

The grubx64.efi image will now attempt to download GRUB2 configuration files in order using the following rules, where the appended value -(something) corresponds to a property or address of the client machine:

(FWPATH)/grub.cfg-(UUID OF NIC)
(FWPATH)/grub.cfg-01-(MAC ADDRESS OF NIC)
(FWPATH)/grub.cfg-(IPv4 OR IPv6 ADDRESS)
(FWPATH)/grub.cfg

  • Note that this GRUB2 information has been copied from the local Linux grub.html manual’s Network section in /usr/share/doc/grub2-common/grub.html because the original manual from gnu.org is frequently inaccessible. Make sure that the package containing the grub.html file has been installed on your PC by dnf install grub2-common.

  • The grub.cfg file is placed in the same directory as the path output by grub-mknetdir hereafter referred to as (FWPATH). Note: Our setup uses FWPATH=/tftpboot/uefi.

  • For the MAC ADDRESS OF NIC value the grub.html manual is missing a leading -01 which we have added here.

The client will only attempt to look up an IPv6 address value once, however, it will try the smaller and smaller parts of IPv4 address multiple times as shown below. The first file in this list which can be downloaded successfully will be used for network booting. This gives flexibility when configuring multiple client computers.

The concrete example below shows what would happen under the IPv4 case:

The GRUB2 bootloader will attempt TFTP download of this list of configuration files in sequential order:

(FWPATH)/grub.cfg-7726a678-7fc0-4853-a4f6-c85ac36a120a
(FWPATH)/grub.cfg-01-52-54-00-ec-33-81        # Note the leading "-01" which is missing in the documentation
(FWPATH)/grub.cfg-0A000082
(FWPATH)/grub.cfg-0A00008
(FWPATH)/grub.cfg-0A0000
(FWPATH)/grub.cfg-0A000
(FWPATH)/grub.cfg-0A00
(FWPATH)/grub.cfg-0A0
(FWPATH)/grub.cfg-0A
(FWPATH)/grub.cfg-0
(FWPATH)/grub.cfg

After GRUB2 has started, files on the TFTP server will be accessible via the (tftp) device.

The TFTP server IP_address can be controlled by changing the (tftp) device name to (tftp,server-ip). Note that this should be changed both in the prefix and in any references to the device name in the configuration file.

Configuring Secure Boot in client setup

If the PXE client system is configured for UEFI Secure_Boot then the PXE boot may likely fail with an error about an invalid signature. See What is UEFI Secure Boot and how it works? and Installation of RHEL8 on UEFI system with Secure Boot enabled fails with error ‘invalid signature’ on vmlinuz.

If you install third party Linux_kernel driver modules the Secure_Boot may block these modules:

Workaround: Disable Secure_Boot from UEFI or BIOS settings. After the OS installation has completed, Secure_Boot may be reenabled and the OS should boot correctly in this mode, unless you build your own custom Linux_kernel due to special device drivers etc.

In some cases it is actually possible to make a successful PXE Secure_Boot installation, see the section on DHCP_server_UEFI_configuration.

You can determine on a running system whether Secure_Boot is enabled or not:

$ mokutil --sb-state

efibootmgr - manipulate the UEFI Boot Manager

efibootmgr is a userspace application used to modify the UEFI Boot Manager. This application can create and destroy boot entries, change the boot order, change the next running boot option, and more.

To show the current boot order:

efibootmgr -v

Some useful command options (see the efibootmgr page):

-n | --bootnext XXXX   set BootNext to XXXX (hex)
-N | --delete-bootnext delete BootNext
-o | --bootorder XXXX,YYYY,ZZZZ,...     explicitly set BootOrder (hex)
-O | --delete-bootorder   delete BootOrder

Configure your network installation server

Install the bootloader image files

Install the boot-image packages on your network installation server:

dnf install grub2-efi-x64 shim-x64

Configure the TFTP service and create a special directory for UEFI bootloader files:

mkdir /var/lib/tftpboot/uefi
ln -s /var/lib/tftpboot /tftpboot

Determine the OS family name for the subfolder in /boot/efi/EFI/ by:

$ grep '^ID=' /etc/os-release
ID="almalinux"        # Or "rocky", "rhel", "centos" or something else

Copy the boot image files from the packages installed above (remember to change their permissions):

cp -p /boot/efi/EFI/BOOT/BOOTX64.EFI /tftpboot/uefi/
cp -p /boot/efi/EFI/<insert OS ID here>/grubx64.efi /tftpboot/uefi/
cp -p /boot/efi/EFI/<insert OS ID here>/shimx64.efi /tftpboot/uefi/
chmod 644 /tftpboot/uefi/BOOTX64.EFI /tftpboot/uefi/grubx64.efi /tftpboot/uefi/shimx86.efi

The shim bootloader

According to the Cambridge Dictionary shim is a small object or piece of material used between two parts of something to make them fit together.

The shimx64.efi is an EFI application that functions as a first-stage bootloader for systems with Secure_Boot enabled. Additionally, shimx64.efi works within the constraints of Secure_Boot, which requires all bootloaders and kernels to be signed with a trusted Microsoft key. It allows the user to individually trust keys provided by various Linux distributions. Further information:

Verify secure boot image signature

This is only optional: You can verify the signature of UEFI secure boot images using the sbverify UEFI secure boot verification tool. First enable the repository:

Install the package:

$ dnf install sbsigntools

Some examples of signatures are:

  • Any Linux shimx64.efi:

    sbverify --list /boot/efi/EFI/rocky/shimx64.efi
warning: data remaining[832368 vs 959224]: gaps between PE/COFF sections?
signature 1
image signature issuers:
 - /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
image signature certificates:
 - subject: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Windows UEFI Driver Publisher
   issuer:  /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
 - subject: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
   issuer:  /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation Third Party Marketplace Root

  • Any Linux BOOTX64.EFI:

    $ sbverify --list /boot/efi/EFI/BOOT/BOOTX64.EFI
signature 1
image signature issuers:
 - /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
image signature certificates:
 - subject: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Windows UEFI Driver Publisher
   issuer:  /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
 - subject: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
   issuer:  /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation Third Party Marketplace Root

  • AlmaLinux system grubx64.efi:

    $ sbverify --list /boot/efi/EFI/almalinux/grubx64.efi
signature 1
image signature issuers:
 - /emailAddress=security@almalinux.org/O=AlmaLinux OS Foundation/CN=AlmaLinux Secure Boot CA
image signature certificates:
 - subject: /emailAddress=security@almalinux.com/O=AlmaLinux OS Foundation/CN=AlmaLinux Secure Boot Signing
   issuer:  /emailAddress=security@almalinux.org/O=AlmaLinux OS Foundation/CN=AlmaLinux Secure Boot CA
 - subject: /emailAddress=security@almalinux.org/O=AlmaLinux OS Foundation/CN=AlmaLinux Secure Boot CA
   issuer:  /emailAddress=security@almalinux.org/O=AlmaLinux OS Foundation/CN=AlmaLinux Secure Boot CA

  • RockyLinux system grubx64.efi:

    $ sbverify --list /boot/efi/EFI/rocky/grubx64.efi
signature 1
image signature issuers:
 - /C=US/ST=Delaware/L=Dover/O=Rocky Enterprise Software Foundation/OU=Release engineering team/CN=Rocky Linux Secure Boot Root CA
image signature certificates:
 - subject: /C=US/ST=Delaware/L=Dover/O=Rocky Enterprise Software Foundation/OU=Release engineering team/CN=Rocky Linux Grub2 Signing Cert 101
   issuer:  /C=US/ST=Delaware/L=Dover/O=Rocky Enterprise Software Foundation/OU=Release engineering team/CN=Rocky Linux Secure Boot Root CA

Setting up the DHCP, TFTP and PXE services

Enable UEFI support in the DHCP server

We use an ISC_DHCP Linux server on EL/RHEL Linux. The ISC_DHCP server has actually been superceded by the ISC_KEA server, but we do not consider it here. On EL Linux ISC_KEA can be installed (in EL8/EL9 from EPEL) with dnf install kea kea-hooks kea-doc kea-keama.

Install the ISC_DHCP packages:

dnf install dhcp-server dhcp-common

To get started with configuration the packages contain an example file /usr/share/doc/dhcp-server/dhcpd.conf.example. It is also recommended to consult examples on the internet, or to read the DHCP_Handbook for complete coverage of the ISC_DHCP server.

Add the following to the configuration file /etc/dhcp/dhcpd.conf in the top (global) section:

# These settings are required for UEFI boot:
option arch code 93 = unsigned integer 16; # RFC4578

The Client System Architecture Type Option 93 (EFI x86-64) is defined in RFC4578.

Add these options only if you need to support MTFTP (Multicast TFTP) as recommended (but undocumented) in many places:

option space PXE;
option PXE.mtftp-ip    code 1 = ip-address;
option PXE.mtftp-cport code 2 = unsigned integer 16;
option PXE.mtftp-sport code 3 = unsigned integer 16;
option PXE.mtftp-tmout code 4 = unsigned integer 8;
option PXE.mtftp-delay code 5 = unsigned integer 8;

In the dhcpd.conf subnet section(s) define the desired UEFI RFC4578 or PXE (legacy) bootloader image types in the /tftpboot/uefi/ subdirectory.

Remember also to Install the bootloader image files. If you have any PXE boot clients with Secure_Boot enabled, you must serve the shimx64.efi first-stage bootloader image in stead of the often-cited BOOTX64.EFI, see the Automated Kickstart installation section. See also the article grubx64_versus_shimx64 and the shim homepage.

You should therefore always serve the shimx64.efi first-stage bootloader image:

# UEFI x86-64 boot (RFC4578 architecture types 7, 8 and 9)
if option arch = 00:07 {
      filename "uefi/shimx64.efi";
} else if option arch = 00:08 {
      filename "uefi/shimx64.efi";
} else if option arch = 00:09 {
      filename "uefi/shimx64.efi";
} else {
      # PXE boot
      filename "pxelinux.0";
}

Note: Other CPU architectures besides x86-64 are listed in the UEFI_specification section 3.5.

The shimx64.efi chainloads grubx64.efi after the Verify_signatures step, and this also works seemlessly on clients that have disabled the Secure_Boot feature.

IMPORTANT:: The shimx64.efi and grubx64.efi bootloader images must be copied from the same Linux OS version as the OS you are trying to install on the client, i.e., the PXE installation Linux_kernel vmlinuz (see below) must have the same signature.

We have not been able to find a way to support multiple OS versions with Secure_Boot clients. Any signature mismatch will cause the installation to fail, since different OS images cannot verify the image signatures of other OSes, for example RHEL versus AlmaLinux versus RockyLinux.

Placing the boot-image file in a subdirectory of the TFTP server’s /tftpboot folder such as /tftpboot/uefi/, will cause the client host PXE boot process to download all further files also from that same subdirectory, so you need to place any other files there.

When you have completed configuring the dhcpd.conf file, open the firewall for DHCP (port 67):

firewall-cmd --add-service=dhcp --permanent
firewall-cmd --reload

and start the DHCP service:

systemctl enable dhcpd
systemctl restart dhcpd

Configure the TFTP service

Your DHCP server should also run a TFTP service for file downloads. Install these packages:

dnf install tftp-server tftp

Copy the service file to make local customizations:

cp /usr/lib/systemd/system/tftp.service /etc/systemd/system/tftp.service

Edit the file /etc/systemd/system/tftp.service to add the in.tftpd options --secure --ipv4:

ExecStart=/usr/sbin/in.tftpd -v --secure --ipv4 /var/lib/tftpboot

Open the firewall for TFTP (port 69):

firewall-cmd --add-service=tftp --permanent
firewall-cmd --reload

and start the service:

systemctl enable tftp
systemctl restart tftp

Download Linux OS boot images

For each EL/RHEL Linux (and other OS) version you should copy Linux boot images to a separate directory on the TFTP server, for example, for AlmaLinux 8.10:

mkdir /var/lib/tftpboot/AlmaLinux-8.10-x86_64/

In this directory create the following Makefile:

OS=almalinux
VERSION=8.10
MIRROR=<your-favorite-mirror>
default:
      @echo "NOTE: Boot images are from ${OS} version ${VERSION}"
      @wget --timestamping ${MIRROR}/${OS}/${VERSION}/BaseOS/x86_64/os/images/pxeboot/initrd.img
      @wget --timestamping ${MIRROR}/${OS}/${VERSION}/BaseOS/x86_64/os/images/pxeboot/vmlinuz

and run a make command to download the boot image files.

Create a grub.cfg file in /tftpboot/uefi/

Please consult the Kickstart installation of EL Linux systems page for a description of automated Linux OS installation.

The uefi/grubx64.efi (or the uefi/BOOTX64.EFI) boot file will be looking for a GRUB2 or Grub configuration file uefi/grub.cfg in the same subdirectory. Create the file /var/lib/tftpboot/uefi/grub.cfg with the contents:

set default="0"
function load_video {
  insmod efi_gop
  insmod efi_uga
  insmod video_bochs
  insmod video_cirrus
  insmod all_video
}
load_video
set gfxpayload=keep
insmod net
insmod efinet
insmod tftp
insmod gzio
insmod part_gpt
insmod ext2
set timeout=60
menuentry 'AlmaLinux 8.10 minimal Kickstart' --class centos --class gnu-linux --class gnu --class os --unrestricted {
  # Note: IPv6 disable during initial boot:
  linuxefi (tftp)/AlmaLinux-8.10-x86_64/vmlinuz ip=dhcp inst.ks=nfs:nfsvers=3:10.10.10.3:/u/kickstart/ks-rockylinux-8-minimal-x86_64.cfg ipv6.disable=1
  initrdefi (tftp)/AlmaLinux-8.10-x86_64/initrd.img
}

Note: Change the IP address 10.10.10.3 to that of your local NFS server.

Additional menu entries may be appended to the above, for example:

menuentry 'AlmaLinux 9.6 minimal Kickstart' --class centos --class gnu-linux --class gnu --class os --unrestricted {
  linuxefi (tftp)/AlmaLinux-9.6-x86_64/vmlinuz ip=dhcp inst.ks=nfs:nfsvers=3:10.10.10.3:/u/kickstart/ks-rockylinux-9-minimal-x86_64.cfg ipv6.disable=1
  initrdefi (tftp)/AlmaLinux-9.6-x86_64/initrd.img
}

It is useful to have a grub.cfg menu item from the TFTP server which allows to boot the system from an existing OS installation on disk. This should be the default menu item. To boot a system with grubx64.efi (provided by the grub2-efi-x64 package) in the 1st partition of the first disk hd0:

menuentry 'Useless: Boot from local disk' {
  # Undocumented "exit" command.  Returns to BIOS boot menu on Dell 9020
  exit
}

If there are multiple disks in the client computer, Grub will name them as hd0, hd1, hd2, etc. It seems that the numbering of such disks may vary, and if the OS installation is suddenly in disk hd1 in stead of hd0, it is useful to define a fallback boot menu item as in this example:

set default=0
set fallback=1
menuentry 'Boot from local disk hd0' {
 set root=(hd0,1)
 chainloader /efi/centos/grubx64.efi
}
menuentry 'Boot from local disk hd1' {
 set root=(hd1,1)
 chainloader /efi/centos/grubx64.efi
}

Automated network installation with pxeconfig

You can automate the PXE network booting process completely using the pxeconfig_toolkit written by Bas van der Vlies. Download the pxeconfig_toolkit and read the pxeconfig_installation page.

NOTE: We assume throughout the use of client UEFI booting, since the old BIOS booting is more or less deprecated.

Installation of pxeconfig on EL Linux

See the pxeconfig_installation page. Configure the default boot method to be UEFI in /usr/local/etc/pxeconfig.conf:

[DEFAULT]
boot_method=uefi

This configures the pxeconfig command to create grub.cfg files in the /tftpboot/uefi/ directory which was created in the Create a grub.cfg file in /tftpboot/uefi/ section.

Having added the port 6611 pxeconfigd service to the services file /etc/services, you must also open port 6611 in the firewall:

firewall-cmd --permanent --zone=public --add-port=6611/tcp --reload

Setup the pxeconfigd service with Systemd. Note that it is pxeconfigd.socket which handles the pxeconfigd service, similar to the normal telnet service, and not the .service file. Remember to set the SELinux context:

restorecon -v /usr/local/sbin/pxeconfigd

Hexadecimally encoded IP-addresses

To understand the client’s hexadecimally encoded IP-address, which the pxeconfig_toolkit manipulates in the server’s /tftpboot/uefi/ directory, we show some examples:

0A018219 decodes as 10.1.130.25

You can use the gethostip command from the syslinux package to convert hostnames and IP-addresses to hexadecimal, for example:

$ gethostip -f s001
s001.(domainname) 10.2.130.21 0A028215
$ gethostip -x s001
0A028215

The pxeconfig command

To use pxeconfig you should create any number of configuration files named default.<something> which contain different PXELINUX commands that perform the desired actions, for example, BIOS updates, firmware updates, hardware diagnostics, or network installation. See the above Create a grub.cfg file in /tftpboot/uefi/ section.

Use the pxeconfig command to configure those client nodes that you wish to install (the remaining nodes will simply boot from their hard disk). An example is:

$ pxeconfig c150
Which pxe config file must we use: ?
1 : default.rockylinux-8-sr850v3-x86_64
2 : default.rockylinux-8-x86_64

The pxeconfig command creates soft-links in the /tftpboot/uefi/ directory named as the hexadecimally encoded IP-address of the clients, pointing to one of the files default.*. As designed, the PXE network booting process will download the file given by the hexadecimal IP-address, and hence network installation of the node will take place.

If desired you can remove the soft-link:

$ pxeconfig -r c150

The hexls command

To list the soft links created by pxeconfig use the tool hexls and look for the IP-addresses and/or hostnames. An example output is:

$ hexls /tftpboot/uefi/
default.rockylinux-8-x86_64
grub.cfg
grub.cfg-0A028396 => 10.2.131.150 => c150.nifl.fysik.dtu.dk -> default.rockylinux-8-x86_64

The pxeconfigd service

The pxeconfigd service will remove the hexadecimally encoded IP-address soft-link on the server when contacted on port 6611 by the client node. In order for this to happen, you must create the client’s post-install script to make an action such as this example:

#!/bin/sh
# To be used with the pxeconfigd service:
# Remove the <hex_ipaddr> file from the pxelinux.cfg directory so the client will boot from disk.
telnet <IMAGESERVER> 6611
sleep 1
exit 0

When this script is executed on the node in the post-install phase, the telnet command connects to the pxeconfigd service on the image server, and this daemon will remove the hexadecimally encoded IP-address soft-link in /tftpboot/uefi/ corresponding to the client IP-address which did the telnet connection.