Rebuilding the LCFG master
In normal circumstances you can create a new LCFG master by just adding the
dice/options/lcfg-master-server.h
header to the new profile. After that you would just copy over the data and change the aliases in the DNS (see below for details).
If the LCFG master machine has died things are a bit more awkward, there is no longer a single route for feeding new information into the LCFG slaves. MPU plan to fix this problem by adding a live/hot spare so that failover just requires a change of DNS aliases.
As long as you have at least one LCFG slave you can go onto that server and edit the files directly. All the files are stored somewhere in the
/var/lcfg/conf/server
directory. If you need to do an install from scratch then ensure your changes are on the LCFG slave which has the aliases
lcfghost
and
lcfg
. Note that once you have the LCFG master in service those changes will be overwritten so you will want to note any changes you make and add them via the normal mechanisms (i.e. rfe and svn).
Restoring Data
See
MPUBackupSummary for the information on where the data is mirrored. See
MPURestoringFromBackups for information on how to restore the data safely.
RFE
Getting the rfe data back is fairly simple, just restore the data from the last backup:
[pegasus]root: /usr/local/bin/rsync -av --delete -e ssh /disk/rmirror123/lcfgrfedata/ root@bigfan:/var/rfedata
subversion
First, restore the backup:
[pegasus]root: /usr/local/bin/rsync -av --delete -e ssh /disk/rmirror111/lcfgsvn/ root@bigfan:/var/lcfg/svndump
Inside the
/var/lcfg/svndump
directory you will now have a directory for each subversion repository. In there will be nightly dumps of the subversion database. Select the most recent dump:
$ ls /var/lcfg/svndump/lcfg|tail -1
$ ls /var/lcfg/svndump/source|tail -1
As
root, load the svn databases, referring to the paths to the most recent backups:
$ gunzip -c /var/lcfg/svndump/lcfg/lcfg.2009-09-15.dump.gz | svnadmin load /var/svn/lcfg
$ gunzip -c /var/lcfg/svndump/source/source.2009-09-15.dump.gz | svnadmin load /var/svn/source
these will take quite a while to load. Afterwards you should do
om subversion configure
to ensure the repositories have the correct modes and permissions.
It is also necessary to store a keytab in
/etc/httpd.keytab
for the webdav access. If the new server does not already have an HTTP principal then that will need to be created first. The list of HTTP principals can be discovered with kadmin, for example:
$ /usr/kerberos/sbin/kadmin -p squinney/admin
kadmin: listprincs HTTP/*
If the new machine is not in that list then it can be created like this:
kadmin: ank -randkey HTTP/bigfan.inf.ed.ac.uk
it can then be stored into the keytab file like this:
kadmin: ktadd -k /etc/httpd.keytab HTTP/bigfan.inf.ed.ac.uk@INF.ED.AC.UK
If you are not logged in as
root then you would need to store the keytab into a file somewhere else and then move it as root afterwards. The file must be mode
0400
and it must be owned by the
apache
user.
It is probably necessary to restart apache at this point with
om apacheconf restart
Informatics Inventory
The LCFG headers related to the Informatics Inventory database are synchronised from the
infdb host (currently
libra) using the rmirror component. If you have not yet switched the
lcfg-master
alias to the new machine then that address will need adding to the rsync access controls. If you have LCFG running that can be done by editting the source profile for the inf db machine:
!rsync.mentry_lcfgheaders_2 mEXTRA(bigfan.inf.ed.ac.uk)
Once the alias has been changed that entry should be removed.
Alternatively, edit the
lcfgheaders
section of
/var/lcfg/conf/rsync/rsyncd.conf
on the infdb server.
It will probably be necessary to do
om rmirror run
to get the data copied immediately rather than waiting for the regular run which happens at 30 minutes past each hour.
DNS aliases
The LCFG servers and other dependent machines do not refer to the specific machine name of the LCFG master. Instead they refer to
lcfg-master
and
lcfgsvn
. The LCFG subversion repository data is retrieved from
lcfgsvn
. The rfe data, LCFG releases and the Informatics Inventory data are retrieved from
lcfg-master
.
--
StephenQuinney - 17 Sep 2009