[AT] - How to configure a Dell server to provide a console managed by IPMI Serial-over-Lan
Contents
These notes describe how to configure a Dell server at AT so that its console can be managed using IPMI Serial-over-LAN via the School's
conserver
infrastructure.
Important: These instructions apply to AT only; there are separate instructions for configuring a Dell server to provide an IPMI SOL console at either the Informatics Forum or KB. |
The only machines supported in this way are those equipped with Baseboard Management Controllers (BMC's) compliant with IPMI v2.0. In practice, for us, this means '9th Generation' (or above) Dell servers; of the School's current machines, Dell PowerEdge models 1950, 2950 and R900 are suitable candidates.
Any set-up of the BIOS and BMC will necessarily be machine- and manufacturer-specific; these notes currently apply
only to Dell PowerEdge's 1950, 2950 and R900, but they will be kept up-to-date as new machines come into use.
Dell PowerEdge 1950 and 2950's are both fitted with two network interface cards (NICs). In what follows, we are assuming that the machine is connected to the network via its primary network interface card, NIC1, (it
may also be connected via NIC2), and that the port to which NIC1 is connected carries the Informatics 'wire' appropriate to the BMC.
1. BIOS configuration
Irrespective of whether or not the machine has already been installed as a DICE server, certain changes have to be made at the BIOS level in order that the BIOS console is redirected to the appropriate serial port.
Turn on or restart the machine, select
F2
, wait for the
System Setup screen to appear, navigate to the
Serial Communication field, and make the following changes:
Older Dell servers:
Serial Communication |
DEFAULT: |
CHANGE TO: |
Serial Communication |
On without Console Redirection |
On with Console Redirection via COM2 |
External Serial Connector |
COM1 |
COM1 |
Failsafe Baud Rate |
115200 |
115200 |
Remote Terminal Type |
VT100/VT220 |
VT100/VT220 |
Redirection After Boot |
Enabled |
Disabled |
More recent Dell servers:
Serial Communication |
DEFAULT: |
CHANGE TO: |
Serial Communication |
On without Console Redirection |
On with Console Redirection via COM2 |
External Port Address |
Serial Device1=COM1,Serial Device2=COM2 |
Serial Device1=COM1,Serial Device2=COM2 |
External Serial Connector |
Serial Device1 |
Serial Device1 |
Failsafe Baud Rate |
115200 |
115200 |
Remote Terminal Type |
VT100/VT220 |
VT100/VT220 |
Redirection After Boot |
Enabled |
Disabled |
Notes:
- Setting BIOS console redirection via COM2 (rather than COM1) is necessary, since that's the serial port to which the BMC is internally connected.
- Although it would seem to make sense to set the external serial connector to COM2 (so that a 'normal' serial console could be connected to the physical serial port if, for example, IPMI stopped working) doing so causes a problem for some models of Dell server. So the recommendation here is to leave that set to COM1.
2. Initial Baseboard Management Controller configuration
The BMC itself also needs to be configured. This can be done either pre-boot from the BIOS level, or (using
ipmitool
) from the operating system of an machine that has already been installed. Both ways are described here but, since the BIOS changes described in Section 1 require taking the machine down to the BIOS level, it's suggested that the necessary BMC configuration be done at that level as well.
2.1 BIOS-level
Older Dell servers:
Turn on or restart the machine, press
Ctrl-E
when prompted in order to enter the Remote Access Configuration Utility, then make the following changes:
Remote Access Configuration Utility |
DEFAULT: |
CHANGE TO: |
IPMI Over LAN |
Off |
On |
NIC Selection |
Shared |
Shared or Failover (see note 1 below) |
LAN Parameters -> |
IP Address Source |
Static |
DHCP |
MAC Address |
< take a note of this > |
VLAN Enable |
Off |
On |
VLAN ID |
0001 |
493 |
LAN User Configuration -> |
Account User Name: |
root |
root |
Enter Password: |
< something you can remember > |
More recent Dell servers:
If the server has an iDRAC card: turn on or restart the machine, select
F2
, wait for the
System Setup screen to appear, select the 'iDRAC configuration' option and make changes analogous to the above. The necessary NIC failover settings are as follows, but you also need to configure the tagged VLAN ID, and set a password for the root user, as above:
iDRAC Settings |
OPTIONS: |
SELECT: |
Network Settings -> |
Enable NIC |
Disable, Enable |
Enable |
NIC Selection |
LOM1, LOM2 |
LOM1 |
Failover -> |
None, LOM2, All LOMs |
All LOMs |
Notes:
- The setting
NIC Selection = Shared
means that the BMC shares the machine's network interface NIC1 with the host operating system - so we're assuming that the machine is indeed connected to the network via NIC1, and that the port to which NIC1 is connected carries the Informatics 'wire' appropriate to the BMC as a tagged VLAN.
The setting NIC Selection = Failover
should be used when two on-board NIC's are being used a bonded pair to provide the machine's network interface. With this setting, the BMC will failover from NIC1 to NIC2 (and vice versa), just as the bonded interface itself will. Again, the ports to which each pair of the bond are connected must carry the Informatics 'wire' appropriate to the BMC as a tagged VLAN.
- VLAN number
493
carries our 'AT Server Management' subnet. Our name for VLAN 493
is ATSOL
, and that is used in section 3 below.
- The password being set here for the BMC's 'root' user is just a temporary one which will later be reset to our 'standard' one - see Section 4. Make it as complex as you like (up to 20 characters); just make sure you can remember it.
2.2 From the operating system
Login to the machine and
nsu
to root.
Load the kernel modules necessary for the OpenIPMI kernel device driver:
/sbin/modprobe ipmi_msghandler
/sbin/modprobe ipmi_si
/sbin/modprobe ipmi_devintf
Issue the command:
ipmitool lan print 1
and take a note of the MAC Address.
Issue the commands:
ipmitool user set password 2 < something you can remember >
ipmitool lan set 1 vlan id 493
ipmitool lan set 1 ipsrc dhcp
ipmitool lan set 1 access on
ipmitool bmc reset cold
Notes:
- The above assumes that the IPMI LAN interface is located on channel 1. Although this is not mandated by the IPMI standard, it appears (from experience) to be the de facto standard on Dell BMC's. To confirm this, issue the command
ipmitool channel info 1
and check that the Channel Medium Type
is 802.3 LAN
.
- The command
ipmitool lan set vlan id
both enables VLAN operation, and sets the VLAN ID. This command is implemented in ipmitool
v1.8.7 or higher: it is, therefore, supported by the version of ipmitool
installed on FC6 and SL5 (v1.8.8).
3. IP address / DNS / DHCP configuration
In order that the BMC can acquire an IP address via DHCP, the usual DNS and DHCP arrangements have to be made:
- Allocate an IP address for the BMC on the AT server management subnet,
192.168.93/24
.
- Ensure that the port to which NIC1 of the host machine is connected carries the corresponding VLAN
ATSOL
(i.e. the VLAN whose 802.1Q tag is 493
) as an additional tagged VLAN. To achieve this, the machine's port entry in the relevant ports file needs to look something like:
port n myserver - AT1 ATSOL
- Add the allocated IP address to the DNS, with the hostname set to
<machinename>.bmc.inf.ed.ac.uk
- Edit the existing LCFG source profile of the machine to which the BMC belongs so that the DHCP resources for the BMC are set appropriately. The resources
bmchostname
and bmcmac
must both be specified: bmchostname
should be the fully-qualified domain name chosen in the preceding step; bmcmac
should be the MAC address identified in Section 2 above.
Example:
[sandilands]idurkacz: rfe -g lcfg/myserver
...[snip]...
dhclient.hostname myserver.inf.ed.ac.uk
dhclient.mac 00:1d:09:6a:c9:b7
!dhclient.cluster mSET(dhcp/all)
/* BMC */
dhclient.bmchostname myserver.bmc.inf.ed.ac.uk
dhclient.bmcmac 00:1d:09:6a:c9:bb
!dhclient.cluster mADD(dhcp/at/consoles)
/* Inventory information */
...[snip]...
Allow time for these DNS and DHCP changes to propagate.
4. Final Baseboard Management Controller configuration
Reboot the machine and confirm that the BMC is active on the network by
ping
'ing it from the AT console server machine
atconsoles
. When the BMC
is active on the network, set its 'root' user password to our standard one, using the
conserver-ipmisetpass
command (which is provided by the
conserver
component) as follows:
You only need to follow the below procedure if you haven't already set the full 20-character password in the BIOS! Plus, see notes about password truncation which currently occurs on modern hadware.
-
ssh
to atconsoles
-
nsu
to root
- Issue the command
/usr/sbin/conserver-ipmisetpass <machinename>.bmc
In step 3, you will be prompted to enter the temporary password you set in Section 2 above.
With this done, the BMC should now be configured in a way that's compatible with our
conserver
set up.
Notes:
- Ensure that the
/usr/sbin/conserver-ipmisetpass <machinename>.bmc
command has successfully executed before you go on to the following steps. If you configure conserver
to provide an IPMI console on a machine before that machine's BMC password has been set, the resulting contention for the machine's BMC makes it difficult subsequently to set the password.
- To put it another way: if the
/usr/sbin/conserver-ipmisetpass <machinename>.bmc
command fails, the most likely cause is that conserver has already been configured to provide an IPMI console for this machine. To rectify this, temporarily remove that console from the configuration of conserver
.
5. Setting LCFG resources
5.1 Machine configuration
Add the following lines to the profile of a machine which is to use an IPMI SOL console:
#define LCFG_OPTS_SERIALCONSOLE_TTY 1
#define LCFG_OPTS_SERIALCONSOLE_BAUD 57600
#include <dice/options/serialconsole.h>
Notes:
- The
LCFG_OPTS_SERIALCONSOLE_TTY 1
macro definition puts the machine's console on ttyS1
(i.e. COM2
) rather than the standard ttyS0
, and LCFG_OPTS_SERIALCONSOLE_BAUD 57600
sets the baud rate of the serial console to 57600
rather than the standard 9600
. The significance is that COM2
is the serial port which is internally connected to the BMC, and 57600
is the default baud rate of the BMC.
- On some more recent Dells, the baud rate needs to be set to 115200 in the header instead of 57600. So far, this applies to the following models:
- R210 II (BMC: iDRAC 6 Express)
- R310
- R320
- R610
- R720
5.2 conserver
configuration
Edit the
live/console_server.h
header, and add the target machine's hostname (
not the hostname of its BMC) to the next available free IPMI SOL 'slot' of
marriner
, the console server which takes care of IPMI SOL for the Forum. IPMI SOL 'slot' numbering within our
conserver
set up runs from 01 upwards.
Example:
/********************
* AT Consoles *
********************/
...[snip]...
/* Consoles managed via IPMI SOL */
conserver.consolename_atsol00s01 myserver
Allow time for this live header change to propagate, then test the new console by
ssh
'ing to any console server and running
console <machinename>
.
6. Using the console
A console implemented via IPMI SOL is used in exactly the same way as any other console managed by
conserver
here - so refer to the
local conserver documentation.
In particular, note that console logs can be found in the
/var/consoles
directory of the corresponding console server machine, and that a SysRq break is sent to the target machine using the sequence
Ctrl-] Ctrl-] l 0
(that's lower-case L, followed by a zero).
7. Remote power control
If a machine has been set up as above, then its power supply can be remotely controlled from the AT console server
atconsoles
via IPMI. To do this, use the
conserver-ipmipower
command (a wrapper to
ipmitool
) which is provided by the
conserver
component:
-
ssh
to atconsoles
-
nsu
to root
- Issue the command
/usr/sbin/conserver-ipmipower <bmchostname> <power command>
Examples:
[verte]root: /usr/sbin/conserver-ipmipower
Usage: conserver-ipmipower [-hv] <bmchostname> <power command>
Wrapper for 'ipmitool -H <bmchostname> chassis power <power command>'
Options: -h This help
-v Verbose
chassis power Commands: status, on, off, cycle, reset, diag, soft
[verte]root: /usr/sbin/conserver-ipmipower myserver.bmc status
Chassis Power is on
The effect of the various power commands is as follows:
status |
Show current power status |
on |
Power up chassis, and reboot |
off |
Power down chassis (Note that this does not initiate a clean shutdown: the effect is the same as abruptly hitting the front-panel power button.) |
cycle |
Power cycle chassis (Has the effect: abrupt power off, 1 second wait, power on.) |
soft |
Initiate a soft shutdown and power-off of a live machine (Has the effect of shutdown -h now .) |
Note: The setting of the machine's BIOS
AC Power Recovery
parameter has no effect on any of the above behaviour. That setting affects the behaviour of the system when mains power is restored to the system
as a whole (i.e. through the incoming power cables); but the fact that the IPMI interface is functional at all implies that mains power is being applied to the system. So, for example, a
soft shutdown, followed by a
on, should always result in a reboot.
For more information on the details of the available power commands,
man ipmitool
.
8. Further information
- Dell OpenManage Baseboard Management Controller Utilities User's Guide
- School of Informatics guide to using (ssh)console and conserver
- IPMI home page
-
man ipmitool
--
IanDurkacz - 17 Feb 2010