Moving the LCFG Master to a new machine
This is how we plan to move the LCFG master from
steen to
canopus
Preparation
- Edit
live/tartarus-server.h
to allow access from canopus for the dice-check script:
TARTARUS_ADD_API_PRINCIPAL("dicecheck/canopus.inf.ed.ac.uk\@INF.ED.AC.UK")
- Edit
dice/options/dice/options/tartarus-server.h
to allow access from canopus to the lcfg headers generated by tartarus:
rsync.mentry_lcfghdrs_2 hosts allow=steen.inf.ed.ac.uk salamanca.inf.ed.ac.uk canopus.inf.ed.ac.uk
(This will also require a temporary live modification to the tartarus server profile to get immediate access to the data.)
- Edit the lcfg profile for steen to allow access to all rsync modules from canopus:
!rsync.mentry_autocheckout_hallow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_infinv_allow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_lcfgdefaults_allow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_lcfginf_allow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_lcfgpreviousrelease_allow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_lcfgreleases_allow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_lcfgrfedata_allow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_lcfgstablerelease_allow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_lcfgsvn_hallow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_lcfgtest_allow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_lcfgtestingrelease_allow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_svndatadir_hallow mADD(canopus.inf.ed.ac.uk)
Beforehand
Every day or so until the day of the move:
- Copy over the rfe data from steen to canopus:
-
rsync -av --delete steen::rfedata/ /var/rfedata/
- Copy over the dumps of the subversion repositories:
-
rsync -av --delete steen::lcfgsvn/ /var/lcfg/svndump/
- Copy
/var/svn
:
-
rsync -av --delete steen::svndatadir/ /var/svn/
- Copy over the testing and stable releases:
-
rsync -av --delete steen::lcfgtestingrelease /var/lcfg/releases/testing/
-
rsync -av --delete steen::lcfgstablerelease /var/lcfg/releases/stable/
- Copy the releases archive:
-
rsync -av --delete steen::lcfgreleases /var/cache/lcfgreleases/
On the day
- Mail lcfg-discuss to announce the start of the move.
- Mail cos to announce the start of the move.
- Pacify Nagios for steen and canopus, vega and altair, vole, salamanca, mole and polecat
- Lots of windows on steen and canopus before we start.
nsu
in some of them.
- Change the permissions on the subversion repositories (in
live/lcfg-subversion-authz.h
) so that only MPU members have rw permission for the lcfg subversion repositories. Test this after it's been done.
- Stop the LCFG slave servers:
-
om vega.server stop
-
om altair.server stop
-
om vole.server stop
-
om salamanca.server stop
-
om mole.server stop
-
om polecat.server stop
- Make 'rfe' read-only on both old and new servers. This could be done by stopping the rfe component:
-
om steen.rfe stop
-
om canopus.rfe stop
- Copy over the rfe data from steen to canopus:
-
rsync -av --delete steen::rfedata/ /var/rfedata/
- Dump the subversion repositories on steen - this takes a minute or two:
-
svnadmin dump /var/svn/dice > /var/lcfg/svndump/dice/dice.steen-final.dump
-
svnadmin dump /var/svn/lcfg > /var/lcfg/svndump/lcfg/lcfg.steen-final.dump
-
svnadmin dump /var/svn/source > /var/lcfg/svndump/source/source.steen-final.dump
- Copy over the dumps of the subversion repositories:
rsync -av --delete steen::lcfgsvn/ /var/lcfg/svndump/
Load the repository data into the new repositories. The proper way to do this would be to run svnadmin load
into fresh empty repositories from each steen-final.dump file. However that would take all day, so instead we just use rsync to copy the whole of /var/svn
at once:
-
rsync -av --delete steen::svndatadir/ /var/svn/
For reference this is what the svnadmin load
commands would be if we were using them:
-
svnadmin load /var/svn/dice < /var/lcfg/svndump/dice/dice.steen-final.dump
-
svnadmin load /var/svn/lcfg < /var/lcfg/svndump/lcfg/lcfg.steen-final.dump
-
svnadmin load /var/svn/source < /var/lcfg/svndump/source/source.steen-final.dump
Copy over the testing and stable releases:
-
rsync -av --delete steen::lcfgtestingrelease /var/lcfg/releases/testing/
-
rsync -av --delete steen::lcfgstablerelease /var/lcfg/releases/stable/
Copy the releases archive:
-
rsync -av --delete steen::lcfgreleases /var/cache/lcfgreleases/
Move lcfg-master and lcfgsvn from steen to canopus: rfe dns/inf
Move svn.lcfg.org from steen to canopus: rfe dns/lcfg_org
Update the DNS on each LCFG server:
-
om steen.dns update
-
om canopus.dns update
-
om vega.dns update
-
om altair.dns update
-
om vole.dns update
-
om salamanca.dns update
-
om mole.dns update
-
om polecat.dns update
Make a change to a header to trigger the first autocheckout on the new server. See if something appears in /var/lib/autocheckout/lcfg
. (Alternatively it could be rsynced from the old machine?)
Restart the test slave:
If all is OK, restart diydice:
If all is OK, restart the main LCFG slave servers:
-
om vega.server start
-
om altair.server start
and the others:
-
om salamanca.server start
-
om polecat.server start
Make a test change in a header and make sure it goes all the way through to the affected host(s).
Make a test change via rfe lcfg/
and make sure it goes all the way through to the affected host.
When everything seems OK, edit live/lcfg-subversion-authz.h
to undo the access restrictions imposed earlier.
Restart rfe: om canopus.rfe start
Announce the all clear.
Re-enable subversion dumps on canopus: rfe lcfg/canopus
and remove the cron.objects overrides.
Enable mirror backups on canopus: rfe lcfg/canopus
Remove the lcfg-master.h header from steen: rfe lcfg/steen
Re-enable rfe on steen: om steen.rfe start
Mail lcfg-discuss to announce the completion of the move.
Mail cos to announce the completion of the move.