Rebuilding the LCFG master

In normal circumstances you can create a new LCFG master by just adding the dice/options/lcfg-master-server.h header to the new profile. After that you would just copy over the data and change the aliases in the DNS (see below for details).

If the LCFG master machine has died things are a bit more awkward, there is no longer a single route for feeding new information into the LCFG slaves. TODO: MPU plan to fix this problem by adding a live/hot spare so that failover just requires a change of DNS aliases.

As long as you have at least one LCFG slave you can go onto that server and edit the files directly. All the files are stored somewhere in the /var/lcfg/conf/server directory. If you need to do an install from scratch then ensure your changes are on the LCFG slave which has the aliases lcfghost and lcfg. Note that once you have the LCFG master in service those changes will be overwritten so you will want to note any changes you make and add them via the normal mechanisms (i.e. rfe and svn).

Restoring Data

See MPUBackupSummary for the information on where the data is mirrored. See MPURestoringFromBackups for information on how to restore the data safely.

RFE

Getting the rfe data back is fairly simple, just restore the data from the last backup:

[pegasus]root: /usr/local/bin/rsync -av --delete -e ssh /disk/rmirror123/lcfgrfedata/ root@bigfan:/var/rfedata

subversion

First, restore the backup:

[pegasus]root: /usr/local/bin/rsync -av --delete -e ssh /disk/rmirror111/lcfgsvn/ root@bigfan:/var/lcfg/svndump

Inside the /var/lcfg/svndump directory you will now have a directory for each subversion repository. In there will be nightly dumps of the subversion database. Select the most recent dump:

$ ls /var/lcfg/svndump/lcfg|tail -1
$ ls /var/lcfg/svndump/source|tail -1

As root, load the svn databases, referring to the paths to the most recent backups:

$ gunzip -c /var/lcfg/svndump/lcfg/lcfg.2009-09-15.dump.gz | svnadmin load /var/svn/lcfg
$ gunzip -c /var/lcfg/svndump/source/source.2009-09-15.dump.gz | svnadmin load /var/svn/source

these will take quite a while to load. Afterwards you should do om subversion configure to ensure the repositories have the correct modes and permissions.

It is also necessary to store a keytab in /etc/httpd.keytab for the webdav access. If the new server does not already have an HTTP principal then that will need to be created first. The list of HTTP principals can be discovered with kadmin, for example:

$ /usr/kerberos/sbin/kadmin -p squinney/admin
kadmin:  listprincs HTTP/*

If the new machine is not in that list then it can be created like this:

kadmin: ank -randkey HTTP/bigfan.inf.ed.ac.uk

it can then be stored into the keytab file like this:

kadmin: ktadd -k /etc/httpd.keytab HTTP/bigfan.inf.ed.ac.uk@INF.ED.AC.UK

If you are not logged in as root then you would need to store the keytab into a file somewhere else and then move it as root afterwards. The file must be mode 0400 and it must be owned by the apache user.

It is probably necessary to restart apache at this point with om apacheconf restart

If the LCFG master has died it will be very difficult to get the necessary SSL certificates generated in the usual automated manner. The best solution is to plough on and get the LCFG master working and then come back to the web frontend later.

For the lcfg config repository you need to populate the autocheckout directory which feeds the live data to the LCFG slaves. As root do this:

$ cd /tmp
$ svn co file:///var/svn/lcfg/core lcfg
$ echo >> /tmp/lcfg/include/dice/defaults.h
$ svn commit -m "forced update" 

That commit will take a while, now check that the directory /var/lib/autocheckout/lcfg/lcfg contains branches, core and live sub-directories.

Informatics Inventory

The LCFG headers related to the Informatics Inventory database are synchronised from the infdb host (currently libra) using the rmirror component. If you have not yet switched the lcfg-master alias to the new machine then that address will need adding to the rsync access controls. If you have LCFG running that can be done by editting the source profile for the inf db machine:

!rsync.mentry_lcfgheaders_2 mEXTRA(bigfan.inf.ed.ac.uk)

Once the alias has been changed that entry should be removed.

Alternatively, edit the lcfgheaders section of /var/lcfg/conf/rsync/rsyncd.conf on the infdb server.

It will probably be necessary to do om rmirror run to get the data copied immediately rather than waiting for the regular run which happens at 30 minutes past each hour.

stable/testing releases

These need to be reinstalled from the subversion, this can be done with the normal release-scripts once they are editted to contain the name of the new lcfg-master. TODO: Switch to a single config file for the release scripts to avoid having to alter lots of scripts.

DNS aliases

The LCFG servers and other dependent machines do not refer to the specific machine name of the LCFG master. Instead they refer to lcfg-master and lcfgsvn. The LCFG subversion repository data is retrieved from lcfgsvn. The rfe data, LCFG releases and the Informatics Inventory data are retrieved from lcfg-master.

-- StephenQuinney - 17 Sep 2009