Rebuilding the LCFG master

In normal circumstances you can create a new LCFG master by just adding the dice/options/lcfg-master-server.h header to the new profile. After that you would just copy over the data and change the aliases in the DNS (see below for details).

If the LCFG master machine has died things are a bit more awkward, there is no longer a single route for feeding new information into the LCFG slaves. MPU plan to fix this problem by adding a live/hot spare so that failover just requires a change of DNS aliases.

As long as you have at least one LCFG slave you can go onto that server and edit the files directly. All the files are stored somewhere in the /var/lcfg/conf/server directory. If you need to do an install from scratch then ensure your changes are on the LCFG slave which has the aliases lcfghost and lcfg. Note that once you have the LCFG master in service those changes will be overwritten so you will want to note any changes you make and add them via the normal mechanisms (i.e. rfe and svn).

Restoring Data

See MPUBackupSummary for the information on where the data is mirrored. See MPURestoringFromBackups for information on how to restore the data safely.

RFE

Getting the rfe data back is fairly simple, just restore the data from the last backup:

[pegasus]root: /usr/local/bin/rsync -av --delete -e ssh /disk/rmirror123/lcfgrfedata/ root@bigfan:/var/rfedata

subversion

First, restore the backup:

[pegasus]root: /usr/local/bin/rsync -av --delete -e ssh /disk/rmirror111/lcfgsvn/ root@bigfan:/var/lcfg/svndump

Inside the /var/lcfg/svndump directory you will now have a directory for each subversion repository. In there will be nightly dumps of the subversion database. Select the most recent dump:

$ ls /var/lcfg/svndump/lcfg|tail -1
$ ls /var/lcfg/svndump/source|tail -1

As root, load the svn databases, referring to the paths to the most recent backups:

$ gunzip -c /var/lcfg/svndump/lcfg/lcfg.2009-09-15.dump.gz | svnadmin load /var/svn/lcfg
$ gunzip -c /var/lcfg/svndump/source/source.2009-09-15.dump.gz | svnadmin load /var/svn/source

these will take quite a while to load. Afterwards you should do om subversion configure to ensure the repositories have the correct modes and permissions.

It is also necessary to store a keytab in /etc/httpd.keytab for the webdav access. If the new server does not already have an HTTP principal then that will need to be created first. The list of HTTP principals can be discovered with kadmin, for example:

$ /usr/kerberos/sbin/kadmin -p squinney/admin
kadmin:  listprincs HTTP/*

If the new machine is not in that list then it can be created like this:

kadmin: ank -randkey HTTP/bigfan.inf.ed.ac.uk

it can then be stored into the keytab file like this:

kadmin: ktadd -k /etc/httpd.keytab HTTP/bigfan.inf.ed.ac.uk@INF.ED.AC.UK

If you are not logged in as root then you would need to store the keytab into a file somewhere else and then move it as root afterwards. The file must be mode 0400 and it must be owned by the apache user.

It is probably necessary to restart apache at this point with om apacheconf restart

Informatics Inventory

The LCFG headers related to the Informatics Inventory database are synchronised from the infdb host (currently libra) using the rmirror component. If you have not yet switched the lcfg-master alias to the new machine then that address will need adding to the rsync access controls. If you have LCFG running that can be done by editting the source profile for the inf db machine:

!rsync.mentry_lcfgheaders_2 mEXTRA(bigfan.inf.ed.ac.uk)

Once the alias has been changed that entry should be removed.

Alternatively, edit the lcfgheaders section of /var/lcfg/conf/rsync/rsyncd.conf on the infdb server.

It will probably be necessary to do om rmirror run to get the data copied immediately rather than waiting for the regular run which happens at 30 minutes past each hour.

DNS aliases

The LCFG servers and other dependent machines do not refer to the specific machine name of the LCFG master. Instead they refer to lcfg-master and lcfgsvn. The LCFG subversion repository data is retrieved from lcfgsvn. The rfe data, LCFG releases and the Informatics Inventory data are retrieved from lcfg-master.

-- StephenQuinney - 17 Sep 2009