TWiki> DICE Web>PxeService (revision 6)EditAttach

The DICE PXE service

The PXE service actually consists of three separate services: the main DICE dhcpd server, a TFTP server and an NFS server. Mainly it is used to drive the LCFG installer for DICE machines but it also provides access to utilities for testing memory, wiping hard disks and detecting hardware.

The DHCP Service

The dhcpd server is managed by the Infrastructure Unit. The most important thing is that it tells the client the address of the PXE (i.e. TFTP) server. Beyond that it is used to deliver a number of pieces of information regarding our DICE/LCFG infrastructure to the LCFG installer.

Most of the information regarding the servers being used is stored in the live/dhcpd_server.h header. Bits that might need to be changed are:

This is the IP address of the TFTP server which is used to serve up the pxelinux binary and configuration. Currently it is hare with a complete backup installation on wildcat which can be switched to immediately.

This is the default value for the path to the NFS root. It is the combination of the IP address for the NFS server and the exported path. The NFS server is currently on hare. There is no live backup - currently it would need to be regenerated which is a fairly simple process (see later). This can be altered on a per-client basis via the dhclient.hostrootpath so try there first.

The URL for the LCFG profiles directory. It is unlikely that this will need to be changed except in an emergency when the machine hosting lcfghost is down and the DNS and/or the LCFG profiles cannot be altered.

The URL for the DIY DICE LCFG profiles directory. Again, it is unlikely that this will need to be changed except in an emergency when the machine hosting diydice is down and the DNS and/or the LCFG profiles cannot be altered.

The TFTP service

Bringing up the PXE TFTP service on a new host is relatively simple. The LCFG source profile needs to include dice/options/pxeserver.h

Once the changes have made it to the LCFG client on the new host, it is just a case of:

% om updaterpms run
% om pxeserver start
% om xinetd restart

Note that starting the pxeserver component takes quite a while as it has to generate about 1100 configuration files.

The tftpd serves files from the /tftpboot directory. The important bits this contains are:

  • A set of kernel-install-* directories, one per supported platform. There is a separate package for each of these. See LCFGLinuxInstall for details on how they are made.
  • The pxelinux.cfg directory which is populated by the pxeserver component.
  • A set of pxelinux modules from the syslinux-tftpboot RPM
  • The dban kernel from the dban RPM

All the per-host data in the LCFG pxeserver component resources, which are used to generated the pxelinux config files, comes from the pxeclient component in each client profile via a spanning map.

There are also two pieces of manually maintained data on the tftp server. These are not crucial and can be safely ignored as they are only used by the hardware detection module to map PCI IDs to kernel modules. To add them as well do something like this (as root):

$ cp /lib/modules/2.6.18-128.7.1.el5.inf.1/modules.pcimap /tftpboot/
$ cp /usr/share/hwdata/pci.ids /tftpboot

The TFTP service can be tested like this:

% tftp hare -c get pxelinux.cfg/81D7

That should copy a file named "81D7" into the current directory. If that does not work it is probably a problem with xinetd. I have found that sometimes xinetd can be a nuisance and it might need a good kicking to start tftpd.

The NFS server

The first step to setting up a new PXE NFS root server is to generate the installroot for each supported platform.

For SL5, you would do:

% ssh buildsl5
% nsu
$ /usr/sbin/buildinstallroot --force --root /tmp/sl5-<release name>  --release sl5-stable -o /tmp/sl5.iso
$ cd /tmp
$ tar zcpf /tmp/sl5-<release name>.tar.gz sl5-<release name>
$ exit
% scp /tmp/sl5-<release name>.tar.gz pxerootserver:/tmp
% ssh pxerootserver
% nsu
$ mkdir -p /export/linux/installroot
$ cd /export/linux/installroot
$ tar zxpf /tmp/sl5-<release name>.tar.gz
$ ln -s sl5-<release name> sl5
$ rm /tmp/sl5-<release name>.tar.gz


  • pxerootserver is the name of whatever server you want the PXE NFS root to be served from
  • The release name will be something like 2009072001, just choose the most recent release.
  • Although an ISO is generated it is not used and can be thrown away. However this is an essential part of the process as it makes the PXE install root come out the same as the CD image.
  • This should be repeated for x86_64 for each supported platform. So make an sl5_64 root on the buildsl5_64 machine.
  • It is possible to use a directory other than /export/linux/installroot but it will need to be changed in quite a few places. Easier to just stick with it.
  • The installroot currently weighs in at just over 0.5GB, if there are quite a few supported platforms then clearly a good amount of free space will be required.

Once that is all done add the dice/options/pxe_root_server.h header to the LCFG source profile for the target host.

After the changes have reached the LCFG client of the PXE root server do the following:

% om updaterpms run
% om nfs start

The PXE server LCFG headers

The PXE server headers are spread across 3 levels - lcfg, ed and dice - this is mostly for historical reasons, there are currently no known external users of our lcfg-pxeserver component.

The dice level headers are used to modify the tcpwrappers access appropriately for our network and adding extra modules (e.g. memtest and dban). Anything else which needs to be added which seems to be specific to our network should go here.

The ed level headers are used to add the kernel-pxe-install packages (see LCFGLinuxInstall for details) and set up the appropriate configuration details for each supported platform.

The lcfg level adds the lcfg-pxeserver component package, pulls in the tftp server header and updates syslinux to a modern version.

In an emergency LCFG resources can be overridden via the live/pxeserver.h header.

Testing Changes

There is a backup PXE server (currently wildcat) that can be used to test lcfg-pxeserver changes or new kernels. Talk to Iain Rae nicely and he will usually be happy for you to switch the PXE server for the beowulf wire (which is available in the Forum offices). Switch it by changing the dhcpd.nextserver_O resource in live/dhcpd_server.h.

If there are problems with the LCFG installer you can feed it INITRD_DBG values at the PXE boot prompt which will stop the install script at various points along the way (1 to 9) e.g. INITRD_DBG=4 See the linuxrc script in the lcfg-mkinitramfs package.

-- StephenQuinney - 03 Dec 2009, 07 Feb 2011

-- ChrisCooke updated the PXE server names on 24 Aug 2012

Edit | Attach | Print version | History: r10 | r8 < r7 < r6 < r5 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r6 - 06 Sep 2013 - 15:16:58 - ChrisCooke
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies