How to configure a Scan 3xs server (a rebadged ASUS ECS4000 G3S server) to provide a console managed by IPMI Serial-over-Lan

Contents

These notes describe how to configure a Scan 3xs server (a rebadged ASUS ECS4000 G3S server) - located in either the Informatics Forum, AT or KB - so that its console can be managed using IPMI Serial-over-LAN via the School's conserver infrastructure.

1. Introduction

The ASUS ECS4000 G3S server has three NICs at the rear of the machine, as shown in the following schematic:

+----------------------------------------------------------------------+
|      +-------+    +--------------------+                             |
|      | NIC 3 |    |   Expansion slot   |                             |
|      +-------+    +--------------------+                             |
|      +-------+                                                       |
|      |  USB  |                                                       |
|      +-------+    +-----+ +------------+    +-------+ +-------+      |
|      |  USB  |    | VGA | | Q-Code LED |    | NIC 1 | | NIC 2 |      |
|      +-------+    +-----+ +------------+    +-------+ +-------+      |
|                                                                      |
|      +---------------------------++---------------------------+      |
|      |                           ||                           |      |
|      |         PSU 1             ||         PSU 2             |      |
|      |                           ||                           |      |
|      +---------------------------++---------------------------+      |
|                                                                      |
+----------------------------------------------------------------------+
               REAR PANEL OF AN ASUS ECS4000 G3S SERVER

NICs 1 and 2 are available for use by the machine itself; NIC 3 is a dedicated port for the BMC (which device, as usual, has its own MAC address.) However, the BMC also presents itself - using a different MAC address, on NIC 1. The idea of that is that the BMC can piggy-back on one of the existing network connections in use by the machine itself - which, for example, would save on the total number of network ports in use. However, since the piggy-back setup does not allow for the use of tagged VLANs, it's of no use to us: we insist that all BMCs are located on a private unrouted subnet carried on its own VLAN.

The BMC will, by default, issue DHCP requests on both the piggy-back port NIC 1, and its dedicated port NIC 3 and, as mentioned, each of these requests will be associated with a distinct MAC address. Since we do not want to use the BMC on the piggy-back port NIC 1, any DHCP requests from the BMC on that port will show up as noise and DHCP errors in our various logs, and so we need to disable them.

Our desired configuration is therefore:

1. NICs 1 and 2 to be used by the machine itself; and to be configured on appropriate server VLAN(s).
2. DHCP requests from the BMC to be disabled on NIC 1.
3. NIC 3 to be used by the BMC; and to be configured on the appropriate site-specific BMC VLAN.
4. DHCP requests from the BMC to be enabled on NIC 3.

Note that, out-of-the-box, the BMC is active on the network and, if it acquires an IP address, will immediately accept IPMI commands over the network, with the default privileged (username, password) combination being (admin, admin). To prevent misuse, it is therefore important that the BMC is correctly configured; it must not be left in its default state.

2. BMC configuration

2.1. BIOS configuration

2.1.1 BMC networking

Turn on or restart the machine, and type Del when appropriate to access the BIOS setup screen.

1. Select the Server Mgmt tab.

2. Select BMC network configuration

3. Set the following parameters:

   DM_LAN1:

   Current config address source

   DHCP Mode

   Shared_LAN:

   Current config address source	Static
   IP Address in BMC	0.0.0.0

4. Take a note of the value of Station MAC address in DM_LAN1.

5. Select BMC IPv6 Network Configuration

6. Set the following parameters:

   IPV6 BMC Shared LAN:

   Address Source in BMC	Static
   IP Address in BMC	0:0:0:0:0:0:0:0

7. Press ESC to return to the main screen.

2.1.2 Console redirection

1. Select the Advanced tab.

2. Select Serial Port Console Redirection

3. Set the following parameters:

   COM1:

   Console Redirection

   Enabled

   COM2:

   Console Redirection

   Disabled

4. For COM1, select Console Redirection Settings

   Set the following parameters (all of which - apart from Terminal Type, should be the default settings):

   Terminal Type	VT100+
   Bits per second	57600
   Data Bits	8
   Parity	None
   Stop Bits	1
   Flow Control	Hardware RTS/CTS
   VT-UTF8 Combo Key Support	Enabled
   Recorder Mode	Disabled
   Legacy OS Redirection Resolution	80x24
   Putty KeyPad	VT100
   Redirection After BIOS POST	Always Enable

5. Press ESC, ESC to return to the main screen; then F10 to Save Changes & Reset.

2.2. IP address / DNS / DHCP configuration

1. Allocate an IP address for the BMC on the appropriate server management subnet, and add that address to the DNS with the corresponding hostname set to <machinename>.bmc.inf.ed.ac.uk. The subnets are as follows:

   Server room	Server management subnet
   Informatics Forum	192.168.68/23
   AT	192.168.93/24
   KB	192.168.94/24

2. Ensure that the port to which NIC 3 of the host machine is connected carries the corresponding VLAN untagged. The VLAN names are as follows:

   Server room	Server management VLAN name	Corresponding VLAN tag
   AT	ATSOL	493
   Informatics Forum	SOL	468
   KB	KBSOL	494

   Example: For a machine in the Informatics Forum, the entry in the relevant ports file would look like:
   port  n myserver.bmc - SOL

3. Add the resources bmchostname and bmcmac to the LCFG profile of the machine: bmchostname should be the fully-qualified domain name chosen in step 1 above; bmcmac should be the Station MAC address identified in Section 2.1.1 above.

   Example:

     ...[snip]...
     /* BMC */
     dhclient.bmchostname            myserver.bmc.inf.ed.ac.uk
     dhclient.bmcmac                 00:1d:09:6a:c9:bb
     !dhclient.cluster               mADD(dhcp/forum/consoles)
     ...[snip]...
   

4. Allow time for the profile to recompile, and for the DNS and DHCP changes to propagate. Once they have propagated, restart the machine and check that the BMC has correctly acquired its configuration by:

   1. ssh'ing to the console server appropriate for the site - either consoles, atconsoles or kbconsoles.
   2. Typing ping <machinename>.bmc.inf.ed.ac.uk. You should get a response; if you don't, sort out the problem before you proceed any further.

2.3. Reconfiguration of the BMC accounts

The accounts configured on the BMC now need to be set up appropriately.

1. ssh to to the console server appropriate for the site - either consoles, atconsoles or kbconsoles.

2. telnet <machinename>.bmc.inf.ed.ac.uk, then proceed as follows (noting that everything is case-sensitive):

     login: admin
     Password: admin
   
     cd map1/accounts/user2
     set username=root
     set group=administrator
     cd ..
     cd user3
     set password=<our 'usual' system password>
     exit
   

   Note: Before you type exit to leave the above session, it would be a good idea to initiate a second telnet session to <machinename>.bmc.inf.ed.ac.uk from the console server appropriate to the site in order to check that the password for the admin username (i.e. user ID 3) has indeed been set correctly to our 'usual' one!

3. Back on your ssh session on the console server appropriate for the site (either consoles, atconsoles or kbconsoles):

   1. nsu to root
   2. Issue the command /usr/sbin/conserver-ipmisetpass <machinename>.bmc

   This command changes the password of the BMC's root account to our standard one. When it runs, you will prompted for the current default password of the root account: it is Operator.

3. Setting LCFG resources

3.1 Machine configuration

Add the following line to the profile of a machine which is to use an IPMI SOL console:

  #include <dice/options/serialconsole.h>

Note that this configuration is - unlike that for a Dell server using an IPMI SOL console - exactly the same as for a standard directly-attached serial console. That is, the baud rate and serial interface to be specified are 9600 and ttyS0 (i.e. COM1) respectively.

3.2 conserver configuration

Edit the live/console_server.h header, and add the target machine's hostname (not the hostname of its BMC) to the next available free IPMI SOL 'slot' for the relevant site.

Example:

  /********************
   *  Forum Consoles  *
   ********************/
   ...[snip]...

  /* Consoles managed via IPMI SOL */

   ...[snip]...
  conserver.consolename_srsol00s42      myserver
   ...[snip]...  

Allow time for this live header change to propagate, then test the new console by ssh 'ing to any console server and running console <machinename>.

4. Further information

1. Manuals for the ASUS ESC4000 G3S Server series

-- IanDurkacz - 17 Mar 2017