The DICE PXE service

The PXE service actually consists of three separate services: the main DICE dhcpd server, a kernel/initramfs server (TFTP/http) and an NFS server. Mainly it is used to drive the LCFG installer for DICE machines but it also provides access to utilities for testing memory, wiping hard disks and detecting hardware.

The initial fetch of the pxelinux binary (which is either lpxelinux.0 or efi64/syslinux.efi) is done over TFTP as that's usually the only protocol supported by the network card firmware. However the modern pxelinux binary also contains support for loading any additional files via HTTP, and we use that facility to subsequently fetch the kernel/initramfs.

The TFTP and HTTP services currently run on regulus with a complete backup installation on maia which can be switched to immediately (and which can also be used for testing awkward changes). There is also aldebaran which is a VM on the develop release which is available for development work.

The NFS server is currently on regulus (check pxerootserver CNAME to be sure).

Note that these machines are also the RPM package cache servers (running squid) and thus it is intentional that one is in the Forum and one is in AT.

The DHCP service

The dhcpd server is managed by the Infrastructure Unit. The most important thing is that it tells the client the address of the PXE (i.e. TFTP) server. Beyond that it is used to deliver a number of pieces of information regarding our DICE/LCFG infrastructure to the LCFG installer.

Most of the information regarding the servers being used is stored in the live/dhcpd_server.h header. Bits that might need to be changed are:

This is the IP address of the TFTP server which is used to serve up the pxelinux binary and configuration.

This is the default value for the path to the NFS root. It is the combination of the IP address for the NFS server and the exported path. This can be altered on a per-client basis via the dhclient.hostrootpath so try there first.

The URL for the LCFG profiles directory. It is unlikely that this will need to be changed except in an emergency when the machine hosting lcfghost is down and the DNS and/or the LCFG profiles cannot be altered.

The URL for the DIY DICE LCFG profiles directory. Again, it is unlikely that this will need to be changed except in an emergency when the machine hosting diydice is down and the DNS and/or the LCFG profiles cannot be altered.

The TFTP service

Bringing up the PXE TFTP service on a new host is relatively simple. The LCFG source profile needs to include dice/options/pxeserver.h

Once the changes have made it to the LCFG client on the new host, it is just a case of:

% om updaterpms run
% om pxeserver start
% systemctl start tftp.service
% om apacheconf start

Note that starting the pxeserver component takes quite a while as it has to generate approx 1400 configuration files.

The tftpd serves files from the /var/lib/tftpboot directory. The important bits this contains are:

  • A set of kernel-install-*-pxe directories, one per supported platform. There is a separate package for each of these. See LCFGLinuxInstall for details on how they are made.
  • The pxelinux.cfg directory which is populated by the pxeserver component.
  • A set of pxelinux modules from the syslinux-tftpboot RPM
  • The dban kernel from the dban RPM

All the per-host data in the LCFG pxeserver component resources, which are used to generated the pxelinux config files, comes from the pxeclient component in each client profile via a spanning map.

The TFTP service can be tested like this:

# yum install tftp
% tftp regulus -c get pxelinux.cfg/81D7

That should copy a file named "81D7" into the current directory. If that does not work try restarting the service using systemctl, e.g. systemctl restart tftp.service

The HTTP service

In 2018 we switched to using HTTP to serve the kernel and initramfs files. This makes the downloading of the files much more reliable and also a lot faster than was the case when TFTP was used for the transport.

The associated Apache web server config is fairly straightforward, it is found in /etc/httpd/lcfg.sites.d/pxeweb.conf which in turn includes /etc/httpd/conf.d/pxeweb.conf which sets a list of IP ranges which are allowed access. If a machine cannot download the kernel/initramfs files over http it's probably due to a missing ACL, check the dice/options/pxeweb.h LCFG header for details.

To avoid a conflict with the package cache service the web server listens on a non-standard port (81). You can check the web service by going to with your web browser.

The NFS server

The PXE NFS service is configured via the dice/options/pxe_root_server.h header.

If that header has been added to an LCFG profile for the first time then after the changes have reached the LCFG client of the PXE root server you will need to do the following:

% om updaterpms run
% om nfs start

For each platform supported by the PXE installer (e.g. sl7, sl6_64) there needs to be an associated installroot, which is basically an unpacked directory tree based on the standard CD/DVD installer ISO image. These are stored in the /export/linux/installroot directory (that can be configured by defining the path using DICE_PXE_ROOT_DIR macro).

There is a nightly cron job which automatically copies any new installer images (built after each stable release) from the MPU AFS group space and then does all the necessary unpacking. If you need to run that process manually (e.g. when setting up a new machine) then you can do so using the /usr/lib/lcfg/release-scripts/scripts/update_pxeroot script as root. The script will print out a list of images it has copied (if any), you should now have some installroots available:

cd /export/linux/installroot

Gives a list of directories with platform name prefixes and date stamp suffixes:


Once the installroots are available you will need to create symlinks for the platform names like this:

cd /export/linux/installroot
ln -s sl6_64-2013092301 sl6_64
ln -s sl7-2013092301 sl7

Unless you know that you need a specific release date just select the most recent (according to the date stamp in the directory name).

All the PXE servers should have copies of the most recent install images but be aware they may be in slightly different states as to which image is actually being served (as that's controlled via the symlinks which are handled manually).

You can test the PXE root NFS service from any DICE machine like this:

mount -t nfs pxerootserver:/export/linux/installroot/sl6_64 /mnt

If that mount succeeds then you should be able to see the directories and files of the installroot in the /mnt directory.

Note that only the most recent 3 installroots are kept on the PXE server unless they have an associated symlink in which case they are never deleted. This means that if you're not careful servers can have different installroots available. The current set can be accessed via rsync from the master server (e.g. _regulus):

rsync -av regulus::pxeroots/ /export/linux/installroot/

The PXE server LCFG headers

The PXE server headers are spread across 3 levels - lcfg, ed and dice - this is mostly for historical reasons, there are currently no known external users of our lcfg-pxeserver component.

The dice level headers are used to modify the tcpwrappers access appropriately for our network and adding extra modules (e.g. memtest and dban). Anything else which needs to be added which seems to be specific to our network should go here.

The ed level headers are used to add the kernel-pxe-install packages (see LCFGLinuxInstall for details) and set up the appropriate configuration details for each supported platform.

The lcfg level adds the lcfg-pxeserver component package, pulls in the tftp server header and updates syslinux to a modern version.

In an emergency LCFG resources can be overridden via the live/pxeserver.h header.

Testing Changes

There is a development PXE server (currently aldebaran) that can be used to test lcfg-pxeserver changes or new kernels. That's based in JCMB so choose a wire in that area and switch to it by changing the appropriate dhcpd.nextserver_ resource in live/dhcpd_server.h.

If there are problems with the LCFG installer you can feed it INITRD_DBG values at the PXE boot prompt which will stop the install script at various points along the way (1 to 9) e.g. INITRD_DBG=4 See the linuxrc script in the lcfg-mkinitramfs package.

-- StephenQuinney - 03 Dec 2009, 07 Feb 2011

-- ChrisCooke - 24 Aug 2012 (updated the PXE server names)

-- StephenQuinney - 26 Sep 2013 (revised all the NFS stuff)

-- StephenQuinney - 24 Jan 2019 (update machine names and added notes about web server)

Topic revision: r10 - 16 Apr 2019 - 10:29:42 - IanDurkacz
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies