Rebuilding the LCFG master
In normal circumstances you can create a new LCFG master by just adding the
dice/options/lcfg-master-server.h
header to the new profile. After that you would just copy over the data and change the aliases in the DNS (see below for details).
If the LCFG master machine has died things are a bit more awkward, there is no longer a single route for feeding new information into the LCFG slaves. TODO: MPU plan to fix this problem by adding a live/hot spare so that failover just requires a change of DNS aliases.
As long as you have at least one LCFG slave you can go onto that server and edit the files directly. All the files are stored somewhere in the
/var/lcfg/conf/server
directory. If you need to do an install from scratch then ensure your changes are on the LCFG slave which has the aliases
lcfghost
and
lcfg
. Note that once you have the LCFG master in service those changes will be overwritten so you will want to note any changes you make and add them via the normal mechanisms (i.e. rfe and svn).
Restoring Data
See
MPUBackupSummary for the information on where the data is mirrored. See
MPURestoringFromBackups for information on how to restore the data safely.
RFE
Getting the rfe data back is fairly simple, just restore the data from the last backup:
[pegasus]root: /usr/local/bin/rsync -av --delete -e ssh /disk/rmirror123/lcfgrfedata/ root@bigfan:/var/rfedata
subversion
First, restore the backup:
[pegasus]root: /usr/local/bin/rsync -av --delete -e ssh /disk/rmirror111/lcfgsvn/ root@bigfan:/var/lcfg/svndump
Inside the
/var/lcfg/svndump
directory you will now have a directory for each subversion repository. In there will be nightly dumps of the subversion database. Select the most recent dump:
$ ls /var/lcfg/svndump/lcfg|tail -1
$ ls /var/lcfg/svndump/source|tail -1
As
root, load the svn databases, referring to the paths to the most recent backups:
$ gunzip -c /var/lcfg/svndump/lcfg/lcfg.2009-09-15.dump.gz | svnadmin load /var/svn/lcfg
$ gunzip -c /var/lcfg/svndump/source/source.2009-09-15.dump.gz | svnadmin load /var/svn/source
these will take quite a while to load. Afterwards you should do
om subversion configure
to ensure the repositories have the correct modes and permissions.
It is also necessary to store a keytab in
/etc/httpd.keytab
for the webdav access. If the new server does not already have an HTTP principal then that will need to be created first. The list of HTTP principals can be discovered with kadmin, for example:
$ /usr/kerberos/sbin/kadmin -p squinney/admin
kadmin: listprincs HTTP/*
If the new machine is not in that list then it can be created like this:
kadmin: ank -randkey HTTP/bigfan.inf.ed.ac.uk
it can then be stored into the keytab file like this:
kadmin: ktadd -k /etc/httpd.keytab HTTP/bigfan.inf.ed.ac.uk@INF.ED.AC.UK
If you are not logged in as
root then you would need to store the keytab into a file somewhere else and then move it as root afterwards. The file must be mode
0400
and it must be owned by the
apache
user.
It is probably necessary to restart apache at this point with
om apacheconf restart
If the LCFG master has died it will be very difficult to get the necessary SSL certificates generated in the usual automated manner. The best solution is to plough on and get the LCFG master working and then come back to the web frontend later.
For the lcfg config repository you need to populate the autocheckout directory which feeds the live data to the LCFG slaves. As
root do this:
$ cd /tmp
$ svn co file:///var/svn/lcfg/core lcfg
$ echo >> /tmp/lcfg/include/dice/defaults.h
$ svn commit -m "forced update"
That commit will take a while, now check that the directory
/var/lib/autocheckout/lcfg/lcfg
contains
branches
,
core
and
live
sub-directories.
Informatics Inventory
The LCFG headers related to the Informatics Inventory database are synchronised from the
infdb host (currently
libra) using the rmirror component. If you have not yet switched the
lcfg-master
alias to the new machine then that address will need adding to the rsync access controls. If you have LCFG running that can be done by editting the source profile for the inf db machine:
!rsync.mentry_lcfgheaders_2 mEXTRA(bigfan.inf.ed.ac.uk)
Once the alias has been changed that entry should be removed.
Alternatively, edit the
lcfgheaders
section of
/var/lcfg/conf/rsync/rsyncd.conf
on the infdb server.
It will probably be necessary to do
om rmirror run
to get the data copied immediately rather than waiting for the regular run which happens at 30 minutes past each hour.
stable/testing releases
These need to be reinstalled from the subversion, this can be done with the normal release-scripts once they are editted to contain the name of the new lcfg-master. TODO: Switch to a single config file for the release scripts to avoid having to alter lots of scripts.
DNS aliases
The LCFG servers and other dependent machines do not refer to the specific machine name of the LCFG master. Instead they refer to
lcfg-master
and
lcfgsvn
. The LCFG subversion repository data is retrieved from
lcfgsvn
. The rfe data, LCFG releases and the Informatics Inventory data are retrieved from
lcfg-master
.
--
StephenQuinney - 17 Sep 2009