Dell System Update (dsu)

Dell System Update or dsu automates the updating of firmware on Dell servers. It works on 12G servers and newer. It works on SL7 and (after a fashion) on Ubuntu. There's also a Windows version, which we haven't tried.

Safety first

Please note that using dsu (or any tool) to apply firmware updates can never be a zero-risk approach. MPU recommends a safety-first approach - we do not have access to all possible server models to test these updates. Common sense suggests that you will want to spread your firmware updates over several days/weeks; doing all your critical servers at the same time may end in tears at some point. MPU will make a special announcement for any firmware updates which they consider to be of critical importance and thus requiring immediate application.

What it does

dsu finds out the current firmware levels of the server it's run on. It compares these with what's available in Dell's firmware repository, and offers relevant updates. There are command line options to control whether the updates are installed immediately or downloaded for installation later.

When installation happens, most updates are installed there and then, one at a time. Others - BIOS and PSU updates at least - are staged using the machine's Lifecycle Controller. This means that they will apply at the next reboot, rather than immediately.

How to install dsu

On SL7

You shouldn't need to - it's installed and configured automatically on suitable Dell/DICE servers. (The dice/hw/ header pulls in dice/options/dsu.h.) If dsu is not available on your SL7 Dell server, include dice/options/dsu.h and please also tell the MPU so that they can fix the problem.

On Ubuntu

At the moment you'll need to install it yourself each time you need to run it. Run this as root:
cd /afs/inf.ed.ac.uk/group/mp-unit/dsu
./Systems-Management_Application_8CTK7_LN64_1.9.0_A00.BIN
That should install DSU 1.9.

However DSU 1.9 supports Ubuntu 18.04, and we use 20.04, so you'll need to supply dsu with a couple of outdated libraries. In testing I (Chris) got it working with copies of those libraries as follows:

export LD_PRELOAD="/usr/local/webots/projects/default/resources/sumo/bin/libcrypto.so.1.0.0 /usr/local/webots/projects/default/resources/sumo/bin/libssl.so.1.0.0"
dsu

Import public keys

The first time you run dsu 1.9 it will ask you to import public keys. Answer "y". (And please tell MPU if it fails.)
dsu
DELL EMC System Update 1.9.0
Copyright (C) 2014 DELL EMC Proprietary.
Do you want to import public key(s) on the system (Y/N)? : y
Import Public Key(s)
Importing key 1019CED6 is successful
Importing key 23B66A9D is successful
Importing key 34D8786F is successful
Importing key 46770C59 is successful

How to use dsu

dsu can take several minutes to start up. Be patient. It can be used in several ways:
Just list current firmware levels
dsu -i
Download to a directory
To download updates to a directory for later installation, for example to /var/tmp/dsu, use
dsu --destination-type=CBD --destination-location=/var/tmp/dsu
The update files are downloaded to the named directory, and a script (apply_updates.sh) is provided to install them with later. Having downloaded the updates, you can then boot the machine single user, and run the apply_updates.sh script to install the updates. That way, they can't be interfered with or interrupted by another user or process. This process seems the safest way to use 'dsu' - though it is more time-consuming. This method worked when tested.
Interactive update
dsu lists the firmware updates which can be installed. In interactive mode you choose which updates to install, then start the installation process. Nothing irretrievable happens until you press c for commit.
This method worked when tested. It's probably safe enough to use this method on a stable, quiet machine - but it's not the very safest way of using dsu, because it involves updating firmware while the machine is in multi-user mode, so the firmware updater stands a chance of being interrupted or otherwise interfered with, which could be bad. Also, this method is a good way to get a list of what needs doing.
Non-interactive update
dsu --non-interactive --preview lists all applicable firmware updates then exits.
dsu --non-interactive finds all applicable firmware updates then installs them with no further user input. Only use this if you are absolutely sure of what you are doing.
Make an ISO
Updates can be downloaded and made into an ISO image. The machine can then be booted from that ISO image to install the updates. To make the ISO,
dsu --destination-type=ISO --destination-location=/var/tmp/dsu.iso
In a test, a PowerEdge R710 would not boot from a USB key flashed with such an ISO. If you try this, please let us know how you get on.

Monthly mail report

Dell DICE servers will mail out a report once a month listing all available firmware and BIOS updates for that machine. The report is just the output of the command dsu --non-interactive --preview. This can be prevented by defining DICE_OPTIONS_DSU_INHIBIT_MAIL at the top of a machine's profile.

Quirks and Oddities

dsu can be a little weird at times. Here are some tips and observations. If you come across a new one, let the MPU know. Thanks!
Kernel module loading
Some updates - e.g. BIOS - need to install a kernel module. This is fine unless your machine has disabled kernel module loading. (You do this by including dice/options/kernel-disable-module-loading.h.) If that is the case, you'll need to re-enable kernel module loading before you run dsu. To do this, add something like the following to the top of the machine's LCFG file; wait for the change to reach the machine; then reboot. Then run dsu.
/* REMINDER: kernel module loading has been re-enabled temporarily to help dsu */
#define DICE_OPTIONS_KERNEL_DISABLE_MODULE_LOADING
Please remember to undo this afterwards - remove the lines above then reboot again.
iDRAC updates
When updating iDRAC firmware, disconnect from the serial console. If you don't, the update will fail to apply. You can do such an update using dsu while remotely logged in by ssh. You can also do them using the Lifecycle Controller instead, but only if the Lifecycle Controller offers the same version of update (sometimes it's a version behind dsu).
Doesn't run on the console
dsu has been seen to not work when logged in directly to a server's console. It churns away happily for a few minutes then exits with the message Could not parse the inventory. (This has been observed on a PowerEdge R220 and on a PowerEdge R620.) However dsu worked perfectly well on the same machines when run from a shell obtained via ssh (then nsu). Running its "inventory collector" binary directly produced error messages such as Can't open display and DISPLAY is not set.
Watch out for downgrades!
You may occasionally be offered a downgrade instead of an upgrade. Either be careful - don't just blindly type "a" then "c"! - or try adding the -u or --apply-upgrades-only switch to your dsu command.
RPM installation
dsu installs a couple of RPMs when it runs. They seem harmless, but feel free to remove them afterwards, for example by running updaterpms. They have names like dsucatalog-16.12.00-684PR.noarch and invcol_WF06C_LN64_16.12.200.896_A00-16.12.200.896-WF06C.x86_64.
Safety
The very safest way to do firmware updates is probably to download them to a local directory (see above) then boot the server into single user mode and install the updates. This ensures that only one Dell updater will be running at any one time (reducing the chance of bricking hardware) and that the hardware will be as close to idle as it can be. However, nobody has yet suffered from just running dsu on an idle machine, so that seems fairly safe too.
Logs
An update may occasionally fail with the recommendation that you look in the log for further details. The logs can be found in /var/log/dell.
"Failed to update to standby image"
If an update fails with "Failed to update to standby image" it may be because you're trying to jump too big a gap in firmware versions. In this case upgrading to one or more intermediate firmware versions may help. Firmware updaters can be downloaded form dell.com. Look for .BIN packages for Red Hat Enterprise Linux. Ask Chris or the MPU for help if you have a problem with this.
OS Collector
TLDR: This doesn't seem to be necessary unless Dell asks for it. This isn't firmware, it's a piece of software. When you contact Dell with a problem, and they respond by asking you to send a SupportAssist report to them, the "OS Collector" will fill in the optional "OS and Application Data" part of the report. If it's not installed, the "OS and Application Data" option will be greyed out in the SupportAssist screen. Personal experience suggests that it's not necessary to have it installed unless Dell specifically asks for it.
Failed updates
If an update fails to apply using dsu, it may still apply cleanly via the Lifecycle Controller instead.
Order of applying updates
Dell advises that iDRAC and Lifecycle Controller updates should be applied first, then BIOS, then firmware updates. See iDRAC and Lifecycle Controller - A Recommended Workflow for Performing Firmware Updates on PowerEdge Servers for more details.

Where to find help

  • man dsu provides brief help on the command line options.
  • The homepage explains what's what and provides links to further sources of help.
Topic revision: r27 - 13 Oct 2020 - 08:27:38 - ChrisCooke
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies