[AT] - How to configure a Dell server to provide a console managed by IPMI Serial-over-Lan

Contents

These notes describe how to configure a Dell server at AT so that its console can be managed using IPMI Serial-over-LAN via the School's conserver infrastructure.

The only machines supported in this way are those equipped with Baseboard Management Controllers (BMC's) compliant with IPMI v2.0. In practice, for us, this means '9th Generation' (or above) Dell servers; of the School's current machines, Dell PowerEdge models 1950, 2950 and R900 are suitable candidates.

Any set-up of the BIOS and BMC will necessarily be machine- and manufacturer-specific; these notes currently apply only to Dell PowerEdge's 1950, 2950 and R900, but they will be kept up-to-date as new machines come into use.

Dell PowerEdge 1950 and 2950's are both fitted with two network interface cards (NICs). In what follows, we are assuming that the machine is connected to the network via its primary network interface card, NIC1, (it may also be connected via NIC2), and that the port to which NIC1 is connected carries the Informatics 'wire' appropriate to the BMC.

1. BIOS configuration

Irrespective of whether or not the machine has already been installed as a DICE server, certain changes have to be made at the BIOS level in order that the BIOS console is redirected to the appropriate serial port.

Turn on or restart the machine, select F2, wait for the System Setup screen to appear, navigate to the Serial Communication field, and make the following changes:

Older Dell servers:

More recent Dell servers:

Notes:

1. Setting BIOS console redirection via COM2 (rather than COM1) is necessary, since that's the serial port to which the BMC is internally connected.
2. Although it would seem to make sense to set the external serial connector to COM2 (so that a 'normal' serial console could be connected to the physical serial port if, for example, IPMI stopped working) doing so causes a problem for some models of Dell server. So the recommendation here is to leave that set to COM1.

2. Initial Baseboard Management Controller configuration

The BMC itself also needs to be configured. This can be done either pre-boot from the BIOS level, or (using ipmitool) from the operating system of an machine that has already been installed. Both ways are described here but, since the BIOS changes described in Section 1 require taking the machine down to the BIOS level, it's suggested that the necessary BMC configuration be done at that level as well.

2.1 BIOS-level

Older Dell servers:

Turn on or restart the machine, press Ctrl-E when prompted in order to enter the Remote Access Configuration Utility, then make the following changes:

More recent Dell servers:

If the server has an iDRAC card: turn on or restart the machine, select F2, wait for the System Setup screen to appear, select the 'iDRAC configuration' option and make changes analogous to the above. The necessary NIC failover settings are as follows, but you also need to configure the tagged VLAN ID, and set a password for the root user, as above:

Notes:

1. The setting NIC Selection = Shared means that the BMC shares the machine's network interface NIC1 with the host operating system - so we're assuming that the machine is indeed connected to the network via NIC1, and that the port to which NIC1 is connected carries the Informatics 'wire' appropriate to the BMC as a tagged VLAN.
   The setting NIC Selection = Failover should be used when two on-board NIC's are being used a bonded pair to provide the machine's network interface. With this setting, the BMC will failover from NIC1 to NIC2 (and vice versa), just as the bonded interface itself will. Again, the ports to which each pair of the bond are connected must carry the Informatics 'wire' appropriate to the BMC as a tagged VLAN.
2. VLAN number 493 carries our 'AT Server Management' subnet. Our name for VLAN 493 is ATSOL, and that is used in section 3 below.
3. The password being set here for the BMC's 'root' user is just a temporary one which will later be reset to our 'standard' one - see Section 4. Make it as complex as you like (up to 20 characters); just make sure you can remember it.

2.2 From the operating system

Login to the machine and nsu to root.

Load the kernel modules necessary for the OpenIPMI kernel device driver:

  /sbin/modprobe ipmi_msghandler
  /sbin/modprobe ipmi_si
  /sbin/modprobe ipmi_devintf

Issue the command:

  ipmitool lan print 1

and take a note of the MAC Address.

Issue the commands:

  ipmitool user set password 2 < something you can remember >
  ipmitool lan set 1 vlan id 493
  ipmitool lan set 1 ipsrc dhcp
  ipmitool lan set 1 access on
  ipmitool bmc reset cold

Notes:

1. The above assumes that the IPMI LAN interface is located on channel 1. Although this is not mandated by the IPMI standard, it appears (from experience) to be the de facto standard on Dell BMC's. To confirm this, issue the command ipmitool channel info 1 and check that the Channel Medium Type is 802.3 LAN.
2. The command ipmitool lan set vlan id both enables VLAN operation, and sets the VLAN ID. This command is implemented in ipmitool v1.8.7 or higher: it is, therefore, supported by the version of ipmitool installed on FC6 and SL5 (v1.8.8).

3. IP address / DNS / DHCP configuration

In order that the BMC can acquire an IP address via DHCP, the usual DNS and DHCP arrangements have to be made:

1. Allocate an IP address for the BMC on the AT server management subnet, 192.168.93/24.
2. Ensure that the port to which NIC1 of the host machine is connected carries the corresponding VLAN ATSOL (i.e. the VLAN whose 802.1Q tag is 493) as an additional tagged VLAN. To achieve this, the machine's port entry in the relevant ports file needs to look something like:
   port  n myserver - AT1 ATSOL
3. Add the allocated IP address to the DNS, with the hostname set to <machinename>.bmc.inf.ed.ac.uk
4. Edit the existing LCFG source profile of the machine to which the BMC belongs so that the DHCP resources for the BMC are set appropriately. The resources bmchostname and bmcmac must both be specified: bmchostname should be the fully-qualified domain name chosen in the preceding step; bmcmac should be the MAC address identified in Section 2 above.

Example:

  [sandilands]idurkacz: rfe -g lcfg/myserver

  ...[snip]...
  dhclient.hostname               myserver.inf.ed.ac.uk
  dhclient.mac                    00:1d:09:6a:c9:b7
  !dhclient.cluster               mSET(dhcp/all)

  /* BMC */
  dhclient.bmchostname            myserver.bmc.inf.ed.ac.uk
  dhclient.bmcmac                 00:1d:09:6a:c9:bb
  !dhclient.cluster               mADD(dhcp/at/consoles)

  /* Inventory information */
  ...[snip]...

Allow time for these DNS and DHCP changes to propagate.

4. Final Baseboard Management Controller configuration

Reboot the machine and confirm that the BMC is active on the network by ping 'ing it from the AT console server machine atconsoles. When the BMC is active on the network, set its 'root' user password to our standard one, using the conserver-ipmisetpass command (which is provided by the conserver component) as follows:

You only need to follow the below procedure if you haven't already set the full 20-character password in the BIOS! Plus, see notes about password truncation which currently occurs on modern hadware.

1. ssh to atconsoles
2. nsu to root
3. Issue the command /usr/sbin/conserver-ipmisetpass <machinename>.bmc

In step 3, you will be prompted to enter the temporary password you set in Section 2 above.

With this done, the BMC should now be configured in a way that's compatible with our conserver set up.

Notes:

1. Ensure that the /usr/sbin/conserver-ipmisetpass <machinename>.bmc command has successfully executed before you go on to the following steps. If you configure conserver to provide an IPMI console on a machine before that machine's BMC password has been set, the resulting contention for the machine's BMC makes it difficult subsequently to set the password.
2. To put it another way: if the /usr/sbin/conserver-ipmisetpass <machinename>.bmc command fails, the most likely cause is that conserver has already been configured to provide an IPMI console for this machine. To rectify this, temporarily remove that console from the configuration of conserver.

5. Setting LCFG resources

5.1 Machine configuration

Add the following lines to the profile of a machine which is to use an IPMI SOL console:

  #define LCFG_OPTS_SERIALCONSOLE_TTY 1
  #define LCFG_OPTS_SERIALCONSOLE_BAUD 57600
  #include <dice/options/serialconsole.h>

Notes:

1. The LCFG_OPTS_SERIALCONSOLE_TTY 1 macro definition puts the machine's console on ttyS1 (i.e. COM2) rather than the standard ttyS0, and LCFG_OPTS_SERIALCONSOLE_BAUD 57600 sets the baud rate of the serial console to 57600 rather than the standard 9600. The significance is that COM2 is the serial port which is internally connected to the BMC, and 57600 is the default baud rate of the BMC.
2. On some more recent Dells, the baud rate needs to be set to 115200 in the header instead of 57600. So far, this applies to the following models:
   • R210 II (BMC: iDRAC 6 Express)
   • R310
   • R320
   • R610
   • R720

5.2 conserver configuration

Edit the live/console_server.h header, and add the target machine's hostname (not the hostname of its BMC) to the next available free IPMI SOL 'slot' of marriner, the console server which takes care of IPMI SOL for the Forum. IPMI SOL 'slot' numbering within our conserver set up runs from 01 upwards.

Example:

  /********************
   *  AT Consoles  *
   ********************/
   ...[snip]...

  /* Consoles managed via IPMI SOL */

  conserver.consolename_atsol00s01      myserver  

Allow time for this live header change to propagate, then test the new console by ssh 'ing to any console server and running console <machinename>.

6. Using the console

A console implemented via IPMI SOL is used in exactly the same way as any other console managed by conserver here - so refer to the local conserver documentation.

In particular, note that console logs can be found in the /var/consoles directory of the corresponding console server machine, and that a SysRq break is sent to the target machine using the sequence Ctrl-] Ctrl-] l 0 (that's lower-case L, followed by a zero).

7. Remote power control

If a machine has been set up as above, then its power supply can be remotely controlled from the AT console server atconsoles via IPMI. To do this, use the conserver-ipmipower command (a wrapper to ipmitool) which is provided by the conserver component:

1. ssh to atconsoles
2. nsu to root
3. Issue the command /usr/sbin/conserver-ipmipower <bmchostname> <power command>

Examples:

  [verte]root: /usr/sbin/conserver-ipmipower
  Usage: conserver-ipmipower [-hv] <bmchostname> <power command>
  Wrapper for 'ipmitool -H <bmchostname> chassis power <power command>'
  Options:        -h    This help
                  -v    Verbose
  chassis power Commands: status, on, off, cycle, reset, diag, soft

  [verte]root: /usr/sbin/conserver-ipmipower myserver.bmc status
  Chassis Power is on

The effect of the various power commands is as follows:

Note: The setting of the machine's BIOS AC Power Recovery parameter has no effect on any of the above behaviour. That setting affects the behaviour of the system when mains power is restored to the system as a whole (i.e. through the incoming power cables); but the fact that the IPMI interface is functional at all implies that mains power is being applied to the system. So, for example, a soft shutdown, followed by a on, should always result in a reboot.

For more information on the details of the available power commands, man ipmitool.

8. Further information

1. Dell OpenManage Baseboard Management Controller Utilities User's Guide
2. School of Informatics guide to using (ssh)console and conserver
3. IPMI home page
   • IPMI specifications
     • IPMI v2.0 specifications Document Revision 1.0; Published: Feb 12, 2004
     • IPMI v2.0 Addenda, Errata, and Clarifications Revision 3; Published: February 15, 2006
4. man ipmitool

-- IanDurkacz - 17 Feb 2010