The DICE PXE service

The PXE service actually consists of three separate services: the main DICE dhcpd server, a TFTP server and an NFS server. Mainly it is used to drive the LCFG installer for DICE machines but it also provides access to utilities for testing memory, wiping hard disks and detecting hardware.

The TFTP service is currently running on hare with a complete backup installation on wildcat which can be switched to immediately (and which can also be used for testing awkward changes).

The NFS server is currently on hare. There is no live backup - currently it would need to be regenerated which is a fairly simple process (see later).

Note that these machines are also the RPM package cache servers (running squid) and thus it is intentional that one is in the Forum and one is in AT.

The DHCP Service

The dhcpd server is managed by the Infrastructure Unit. The most important thing is that it tells the client the address of the PXE (i.e. TFTP) server. Beyond that it is used to deliver a number of pieces of information regarding our DICE/LCFG infrastructure to the LCFG installer.

Most of the information regarding the servers being used is stored in the live/dhcpd_server.h header. Bits that might need to be changed are:

This is the IP address of the TFTP server which is used to serve up the pxelinux binary and configuration.

This is the default value for the path to the NFS root. It is the combination of the IP address for the NFS server and the exported path. This can be altered on a per-client basis via the dhclient.hostrootpath so try there first.

The URL for the LCFG profiles directory. It is unlikely that this will need to be changed except in an emergency when the machine hosting lcfghost is down and the DNS and/or the LCFG profiles cannot be altered.

The URL for the DIY DICE LCFG profiles directory. Again, it is unlikely that this will need to be changed except in an emergency when the machine hosting diydice is down and the DNS and/or the LCFG profiles cannot be altered.

The TFTP service

Bringing up the PXE TFTP service on a new host is relatively simple. The LCFG source profile needs to include dice/options/pxeserver.h

Once the changes have made it to the LCFG client on the new host, it is just a case of:

% om updaterpms run
% om pxeserver start
% om xinetd restart

Note that starting the pxeserver component takes quite a while as it has to generate about 1100 configuration files.

The tftpd serves files from the /tftpboot directory. The important bits this contains are:

  • A set of kernel-install-*-pxe directories, one per supported platform. There is a separate package for each of these. See LCFGLinuxInstall for details on how they are made.
  • The pxelinux.cfg directory which is populated by the pxeserver component.
  • A set of pxelinux modules from the syslinux-tftpboot RPM
  • The dban kernel from the dban RPM

All the per-host data in the LCFG pxeserver component resources, which are used to generated the pxelinux config files, comes from the pxeclient component in each client profile via a spanning map.

There are also two pieces of manually maintained data on the tftp server. These are not crucial and can be safely ignored as they are only used by the hardware detection module to map PCI IDs to kernel modules. To add them as well do something like this (as root):

$ cp /lib/modules/2.6.32-358.6.2.el6.x86_64/modules.pcimap /tftpboot/
$ cp /usr/share/hwdata/pci.ids /tftpboot

The TFTP service can be tested like this:

# yum install tftp
% tftp hare -c get pxelinux.cfg/81D7

That should copy a file named "81D7" into the current directory. If that does not work it is probably a problem with xinetd. I have found that sometimes xinetd can be a nuisance and it might need a good kicking to start tftpd.

The NFS server

The PXE NFS service is configured via the dice/options/pxe_root_server.h header.

If that header has been added to an LCFG profile for the first time then after the changes have reached the LCFG client of the PXE root server you will need to do the following:

% om updaterpms run
% om nfs start

For each platform supported by the PXE installer (e.g. sl6, sl6_64) there needs to be an associated installroot, which is basically an unpacked directory tree based on the standard CD/DVD installer ISO image. These are stored in the /export/linux/installroot directory (that can be configured by defining the path using DICE_PXE_ROOT_DIR macro).

There is a nightly cron job which automatically copies any new installer images (built after each stable release) from the MPU AFS group space and then does all the necessary unpacking. If you need to run that process manually (e.g. when setting up a new machine) then you can do so using the /usr/lib/lcfg/release-scripts/scripts/update_pxeroot script as root. The script will print out a list of images it has copied (if any), you should now have some installroots available:

cd /export/linux/installroot

Gives a list of directories with platform name prefixes and date stamp suffixes:


Once the installroots are available you will need to create symlinks for the platform names like this:

cd /export/linux/installroot
ln -s sl6_64-2013092301 sl6_64
ln -s sl6-2013092301 sl6

Unless you know that you need a specific release date just select the most recent (according to the date stamp in the directory name).

You can test the PXE root NFS service from any DICE machine like this:

mount -t nfs pxerootserver:/export/linux/installroot/sl6_64 /mnt

If that mount succeeds then you should be able to see the directories and files of the installroot in the /mnt directory.

The PXE server LCFG headers

The PXE server headers are spread across 3 levels - lcfg, ed and dice - this is mostly for historical reasons, there are currently no known external users of our lcfg-pxeserver component.

The dice level headers are used to modify the tcpwrappers access appropriately for our network and adding extra modules (e.g. memtest and dban). Anything else which needs to be added which seems to be specific to our network should go here.

The ed level headers are used to add the kernel-pxe-install packages (see LCFGLinuxInstall for details) and set up the appropriate configuration details for each supported platform.

The lcfg level adds the lcfg-pxeserver component package, pulls in the tftp server header and updates syslinux to a modern version.

In an emergency LCFG resources can be overridden via the live/pxeserver.h header.

Testing Changes

There is a backup PXE server (currently wildcat) that can be used to test lcfg-pxeserver changes or new kernels. Talk to Iain Rae nicely and he will usually be happy for you to switch the PXE server for the beowulf wire (which is available in the Forum offices). Switch it by changing the dhcpd.nextserver_O resource in live/dhcpd_server.h.

If there are problems with the LCFG installer you can feed it INITRD_DBG values at the PXE boot prompt which will stop the install script at various points along the way (1 to 9) e.g. INITRD_DBG=4 See the linuxrc script in the lcfg-mkinitramfs package.

-- StephenQuinney - 03 Dec 2009, 07 Feb 2011

-- ChrisCooke - 24 Aug 2012 (updated the PXE server names)

-- StephenQuinney - 26 Sep 2013 (revised all the NFS stuff)

Topic revision: r8 - 10 Jun 2014 - 07:18:28 - ChrisCooke
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies