ASUSECS4000G3SIPMISOLConsoleConfiguration < DICE

DICE Web>InfrastructureUnit>Consoles>ASUSECS4000G3SIPMISOLConsoleConfiguration (06 Sep 2017, IanDurkacz) (raw view)EditAttach
<!--
  This is a comment, in the HTML sense of the word, but Wiki directives still work, so you could put
  access control statements here to restrict access to your new topic, and they won't show up in the
  published page.
  Please think carefully when creating a new topic, and use a GoodWikiName.
  Remember that by default all Wiki content is world readable and editable unless you've taken steps
  to limit it.
-->

---+!!How to configure an ASUS ECS4000 !G3S server to provide a console managed by IPMI Serial-over-Lan

---++!!Contents

%TOC%

---++!! 

These notes describe how to configure an ASUS ECS4000 !G3S server (or equallly, a Scan 3xs server) - located in either the Informatics Forum, AT or KB - so that its console can be managed using IPMI Serial-over-LAN via the School's =conserver= infrastructure.

---++1. Introduction

The ASUS ECS4000 !G3S server has three !NICs at the rear of the machine, as shown in the following schematic:

<verbatim>
+----------------------------------------------------------------------+
|      +-------+    +--------------------+                             |
|      | NIC 3 |    |   Expansion slot   |                             |
|      +-------+    +--------------------+                             |
|      +-------+                                                       |
|      |  USB  |                                                       |
|      +-------+    +-----+ +------------+    +-------+ +-------+      |
|      |  USB  |    | VGA | | Q-Code LED |    | NIC 1 | | NIC 2 |      |
|      +-------+    +-----+ +------------+    +-------+ +-------+      |
|                                                                      |
|      +---------------------------++---------------------------+      |
|      |                           ||                           |      |
|      |         PSU 1             ||         PSU 2             |      |
|      |                           ||                           |      |
|      +---------------------------++---------------------------+      |
|                                                                      |
+----------------------------------------------------------------------+
               REAR PANEL OF AN ASUS ECS4000 G3S SERVER
</verbatim>

NICs 1 and 2 are available for use by the machine itself; NIC 3 is a dedicated port for the BMC (which device, as usual, has its own MAC address.) However, the BMC also presents itself - using a _different_ MAC address, on NIC 1. The idea of that is that the BMC can piggy-back on one of the existing network connections in use by the machine itself - which, for example, would save on the total number of network ports in use. However, since the piggy-back setup does _not_ allow for the use of tagged VLANs, it's of no use to us: we insist that all BMCs are located on a private unrouted subnet carried on its own VLAN.

The BMC will, by default, issue DHCP requests on both the piggy-back port NIC 1, and its dedicated port NIC 3 and, as mentioned, each of these requests will be associated with a distinct MAC address. Since we do _not_ want to use the BMC on the piggy-back port NIC 1, any DHCP requests from the BMC on that port will show up as noise and DHCP errors in our various logs, and so we need to disable them.

Our desired configuration is therefore:

   1. NICs 1 and 2 to be used by the machine itself; and to be configured on appropriate server VLAN(s).
   1. DHCP requests from the BMC to be _disabled_ on NIC 1.
   1. NIC 3 to be used by the BMC; and to be configured on the appropriate site-specific BMC VLAN.
   1. DHCP requests from the BMC to be _enabled_ on NIC 3.

Note that, out-of-the-box, the BMC is active on the network and, if it acquires an IP address, will immediately accept IPMI commands over the network, with the default privileged (username, password) combination being (=admin=, =admin=). To prevent misuse, it is therefore important that the BMC is correctly configured; it must _not_ be left in its default state.

---++2. BMC configuration

---+++2.1. BIOS configuration

---+++2.1.1 BMC networking

Turn on or restart the machine, and type =Del= when appropriate to access the BIOS setup screen.

<ol>

<li><p>Select the =Server Mgmt= tab.

<li><p>Select =BMC network configuration=

<li><p>Set the following parameters:

=DM_LAN1=:
%TABLE{cellpadding="3" tablerules="all"}%
|=Current config address source= |=DHCP Mode=|

=Shared_LAN=:
%TABLE{cellpadding="3" tablerules="all"}%
|=Current config address source= |=Static=|
|=IP Address in BMC=|=0.0.0.0=|

<li><p>Take a note of the value of =Station MAC address= in =DM_LAN1=.

<li><p>Press =ESC=

<li><p>Select =BMC IPv6 Network Configuration=

<li><p>Set the following parameters:

=IPV6 BMC Shared LAN=:
%TABLE{cellpadding="3" tablerules="all"}%
|=Address Source in BMC= |=Static=|
|=IP Address in BMC=|=0:0:0:0:0:0:0:0=|

<li><p>Press =ESC= to return to the main screen.

</ol>

---+++2.1.2 Console redirection

<ol>

<li><p>Select the =Advanced= tab.

<li><p>Select =Serial Port Console Redirection=

<li><p>Set the following parameters:

=COM1=:
%TABLE{cellpadding="3" tablerules="all"}%
|=Console Redirection= |=Enabled=|

=COM2=:
%TABLE{cellpadding="3" tablerules="all"}%
|=Console Redirection= |=Disabled=|

<li><p>For =COM1=, select =Console Redirection Settings=
<p>Set the following parameters (all of which - apart from =Terminal Type=, should be the default settings):

%TABLE{cellpadding="3" tablerules="all"}%
| =Terminal Type=|=VT100+=|
|=Bits per second=|=57600=|
|=Data Bits=|=8=|
|=Parity=|=None=|
|=Stop Bits=|=1=|
|=Flow Control=|=Hardware RTS/CTS=|
|=VT-UTF8 Combo Key Support=|=Enabled=|
|=Recorder Mode=|=Disabled=|
|=Legacy OS Redirection Resolution=|=80x24=|
|=Putty !KeyPad=|=VT100=|
|=Redirection After BIOS POST=|=Always Enable=|

<li><p>Press =ESC=

<li><p>For =Legacy Console Redirection=, select =Legacy Console Redirection Settings= 
<p>Set the following parameters:

%TABLE{cellpadding="3" tablerules="all"}%
|=Legacy Serial Redirection Port=||=COM1=|

<li><p>Press =ESC=, =ESC= to return to the main screen; then =F10= to =Save Changes &amp; Reset=.

</ol>

---+++2.2. IP address / DNS / DHCP configuration

<ol>
<li><p>Allocate an IP address for the BMC on the appropriate server management subnet, and add that address to the DNS with the corresponding hostname set to =&lt;machinename&gt;.bmc.inf.ed.ac.uk=. The subnets are as follows:

%TABLE{cellpadding="3" tablerules="all"}%
|*Server room*|*Server management subnet*|
|Informatics Forum|=192.168.68/23=|
|AT|=192.168.93/24=|
|KB|=192.168.94/24=|

<li><p>Ensure that the port to which NIC 3 of the host machine is connected carries the corresponding VLAN _untagged_. The VLAN names are as follows:

%TABLE{cellpadding="3" tablerules="all"}%
|*Server room*|*Server management VLAN name*|*Corresponding VLAN tag*|
|Informatics Forum|=SOL=|468|
|AT|=ATSOL=|493|
|KB|=KBSOL=|494|

_Example:_ For a machine in the Informatics Forum, the entry in the relevant ports file would look like:<br> =port  n myserver.bmc - SOL=

<li><p>Add the resources =bmchostname= and =bmcmac= to the LCFG profile of the machine: =bmchostname= should be the fully-qualified domain name chosen in step 1 above;  =bmcmac= should be the =Station MAC address= identified in Section 2.1.1 above.
<p><em>Example:</em>
<verbatim>
  ...[snip]...
  /* BMC */
  dhclient.bmchostname            myserver.bmc.inf.ed.ac.uk
  dhclient.bmcmac                 00:1d:09:6a:c9:bb
  !dhclient.cluster               mADD(dhcp/forum/consoles)
  ...[snip]...
</verbatim>

<li><p>Allow time for the profile to recompile, and for the DNS and DHCP changes to propagate. Once they _have_ propagated, restart the machine and check that the BMC has correctly acquired its configuration by:
   i. <code>ssh'</code>ing to the console server appropriate for the site - either =consoles=, =atconsoles= or =kbconsoles=.
   i. Typing =ping &lt;machinename&gt;.bmc.inf.ed.ac.uk=. You should get a response; if you don't, sort out the problem before you proceed any further.
</ol>

---+++2.3. Reconfiguration of the BMC accounts

The accounts configured on the BMC now need to be set up appropriately.

<table border="1" cellpadding="3" cellspacing="0">
<tr>
<td>
*Background*

On delivery, the BMC on ASUS ECS4000 !G3S servers come configured with a single active IPMI account, namely:

%TABLE{cellpadding="3" tablerules="all"}%
|*User ID*|*Username*|*Password*|*User Privilege*|
|2|=admin=|=admin=|administrator|

(Note that both username and password are case-sensitive.)

In order that IPMI SOL consoles and power control can be used on these servers within the framework of our existing conserver structure, it's necessary to alter the above configuration so that an IPMI user called =root=, and with ID =2=, exists; and that that account has the same common 20-byte password in use throughout the rest of the systems managed by our =conserver= IPMI framework. In addition, in order that this account can be be used to initiate SOL sessions, it requires 'administrator' privileges.

The easiest way to achieve this is to rename the existing =admin= account. Some BMCs (e.g. those on Supermicro machines) do not allow existing accounts to be renamed - but, fortunately, the BMC on the ASUS servers _does_ allow such renaming.
</td>
</tr>
</table>

<ol>
<li><p><code>ssh</code> to to the console server appropriate for the site - either =consoles=, =atconsoles= or =kbconsoles=.
<li><p>Change the name of BMC user ID =2= on the target machine:<p><code>ipmitool -I lanplus -H &lt;machinename&gt;.bmc -U admin -P admin user set name 2 root</code>
<li><p>Confirm that the name change has been effected:<p><code>ipmitool -I lanplus -H &lt;machinename&gt;bmc -U root -P admin user list</code>
<li><p>Set the password of BMC user ID =2= to our standard 20-byte password:
   i. =nsu= to root
   i. Issue the command =/usr/sbin/conserver-ipmisetpass &lt;machinename&gt;.bmc=
<p>This command changes the password of the BMC's =root= account to our standard one. When it runs, you will prompted for the current _default_ password of the =root= account: it is =admin=.
</ol>

<table border="1" cellpadding="3" cellspacing="0">
<tr>
<td>
<em>Comment:</em> It _ought_ to be equally possible to effect the above account name change from the running OS on the machine to which the BMC belongs, via the =open= channel. However, attempts to do this currently don't work. E.g.

<blockquote>
<code>[<em>machinename</em>]root: ipmitool user list</code><br>
<code>Get User Access command failed (channel 14, user 1): Invalid data field in request</code>
</blockquote>

It seems as if such attempts fail owing to current bugs in <code>ipmitool</code> - see e.g. https://github.com/uebayasi/openbsd-ipmi/issues/10. In any event, at the time of writing, the BMC users must be configured over the network via the =lanplus= channel, as described above.
</td>
</tr>
</table>

---++3. Setting LCFG resources

---+++3.1 Machine configuration

Add the following lines to the profile of a machine which is to use an IPMI SOL console:

<verbatim>
  #define LCFG_OPTS_SERIALCONSOLE_TTY 1
  #define LCFG_OPTS_SERIALCONSOLE_BAUD 57600
  #include <dice/options/serialconsole.h>
</verbatim>

<table border="1" cellpadding="3" cellspacing="0">
<tr>
<td>
<em>Comment:</em> Note that this configuration uses =ttyS1= (i.e. =COM2=) - yet, in section 2.1.2 above, we have configured serial console redirection to =COM1=. This doesn't seem to make sense - but, experimentally, we have confirmed that this suggested configuration works (i.e. it produces redirection of both the =grub= menu, as well as the Linux console - in addition to redirection of the BIOS), whilst other more sensible-looking configurations _don't_ work. There remains some analysis to be done here&nbsp;&#133;
</td>
</tr>
</table>

---+++3.2 =conserver= configuration

Edit the =live/console_server.h= header, and add the target machine's hostname (_not_ the hostname of its BMC) to the next available free IPMI SOL 'slot' for the relevant site.

_Example:_
<verbatim>
  /********************
   *  Forum Consoles  *
   ********************/
   ...[snip]...

  /* Consoles managed via IPMI SOL */

   ...[snip]...
  conserver.consolename_srsol00s42      myserver
   ...[snip]...  
</verbatim>

Allow time for this live header change to propagate, then test the new console by =ssh= 'ing to any console server and running =console &lt;machinename&gt;=.

---++4. Further information

   1. [[https://www.asus.com/uk/Commercial-Servers-Workstations/ESC4000_G3S/HelpDesk_Manual/][Manuals for the ASUS ESC4000 G3S Server series]] 

-- Main.IanDurkacz - 17 Mar 2017
Topic revision: r10 - 06 Sep 2017 - 21:55:42 - IanDurkacz
DICE.ASUSECS4000G3SIPMISOLConsoleConfiguration moved from DICE.Scan3xsASUSECS4000G3SIPMISOLConsoleConfiguration on 21 Mar 2017 - 09:38 by IanDurkacz - put it back
DICE
This is WebLeftBar
Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies