Moving the LCFG Master to a new machine

This is how we plan to move the LCFG master from steen to canopus

Preparation

  • Edit live/tartarus-server.h to allow access from canopus for the dice-check script:

TARTARUS_ADD_API_PRINCIPAL("dicecheck/canopus.inf.ed.ac.uk\@INF.ED.AC.UK")

  • Edit dice/options/dice/options/tartarus-server.h to allow access from canopus to the lcfg headers generated by tartarus:

rsync.mentry_lcfghdrs_2   hosts allow=steen.inf.ed.ac.uk salamanca.inf.ed.ac.uk canopus.inf.ed.ac.uk

(This will also require a temporary live modification to the tartarus server profile to get immediate access to the data.)

  • Edit the lcfg profile for steen to allow access to all rsync modules from canopus:

!rsync.mentry_autocheckout_hallow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_infinv_allow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_lcfgdefaults_allow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_lcfginf_allow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_lcfgpreviousrelease_allow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_lcfgreleases_allow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_lcfgrfedata_allow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_lcfgstablerelease_allow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_lcfgsvn_hallow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_lcfgtest_allow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_lcfgtestingrelease_allow mADD(canopus.inf.ed.ac.uk)
!rsync.mentry_svndatadir_hallow mADD(canopus.inf.ed.ac.uk)

Beforehand

Every day or so until the day of the move:
  • Copy over the rfe data from steen to canopus:
    • rsync -av --delete steen::rfedata/ /var/rfedata/
  • Copy over the dumps of the subversion repositories:
    • rsync -av --delete steen::lcfgsvn/ /var/lcfg/svndump/
  • Copy /var/svn:
    • rsync -av  --delete steen::svndatadir/ /var/svn/
  • Copy over the testing and stable releases:
    • rsync -av --delete steen::lcfgtestingrelease /var/lcfg/releases/testing/
    • rsync -av --delete steen::lcfgstablerelease /var/lcfg/releases/stable/
  • Copy the releases archive:
    • rsync -av --delete steen::lcfgreleases /var/cache/lcfgreleases/

On the day

  • Mail lcfg-discuss to announce the start of the move.
  • Mail cos to announce the start of the move.
  • Pacify Nagios for steen and canopus, vega and altair, vole, salamanca, mole and polecat
  • Lots of windows on steen and canopus before we start. nsu in some of them.
  • Change the permissions on the subversion repositories (in live/lcfg-subversion-authz.h) so that only MPU members have rw permission for the lcfg subversion repositories. Test this after it's been done.
  • Stop the LCFG slave servers:
    • om vega.server stop
    • om altair.server stop
    • om vole.server stop
    • om salamanca.server stop
    • om mole.server stop
    • om polecat.server stop
  • Make 'rfe' read-only on both old and new servers. This could be done by stopping the rfe component:
    • om steen.rfe stop
    • om canopus.rfe stop
  • Copy over the rfe data from steen to canopus:
    • rsync -av --delete steen::rfedata/ /var/rfedata/
  • Dump the subversion repositories on steen - this takes a minute or two:
    • svnadmin dump /var/svn/dice > /var/lcfg/svndump/dice/dice.steen-final.dump
    • svnadmin dump /var/svn/lcfg > /var/lcfg/svndump/lcfg/lcfg.steen-final.dump
    • svnadmin dump /var/svn/source > /var/lcfg/svndump/source/source.steen-final.dump
  • Copy over the dumps of the subversion repositories: rsync -av --delete steen::lcfgsvn/ /var/lcfg/svndump/
  • Load the repository data into the new repositories. The proper way to do this would be to run svnadmin load into fresh empty repositories from each steen-final.dump file. However that would take all day, so instead we just use rsync to copy the whole of /var/svn at once:
    • rsync -av  --delete steen::svndatadir/ /var/svn/
  • For reference this is what the svnadmin load commands would be if we were using them:
    • svnadmin load /var/svn/dice < /var/lcfg/svndump/dice/dice.steen-final.dump
    • svnadmin load /var/svn/lcfg < /var/lcfg/svndump/lcfg/lcfg.steen-final.dump
    • svnadmin load /var/svn/source < /var/lcfg/svndump/source/source.steen-final.dump
  • Copy over the testing and stable releases:
    • rsync -av --delete steen::lcfgtestingrelease /var/lcfg/releases/testing/
    • rsync -av --delete steen::lcfgstablerelease /var/lcfg/releases/stable/
  • Copy the releases archive:
    • rsync -av --delete steen::lcfgreleases /var/cache/lcfgreleases/
  • Move lcfg-master and lcfgsvn from steen to canopus: rfe dns/inf
  • Move svn.lcfg.org from steen to canopus: rfe dns/lcfg_org
  • Update the DNS on each LCFG server:
    • om steen.dns update
    • om canopus.dns update
    • om vega.dns update
    • om altair.dns update
    • om vole.dns update
    • om salamanca.dns update
    • om mole.dns update
    • om polecat.dns update
  • Make a change to a header to trigger the first autocheckout on the new server. See if something appears in /var/lib/autocheckout/lcfg. (Alternatively it could be rsynced from the old machine?)
  • Restart the test slave:
    • om vole.server start
  • If all is OK, restart diydice:
    • om mole.server start
  • If all is OK, restart the main LCFG slave servers:
    • om vega.server start
    • om altair.server start
  • and the others:
    • om salamanca.server start
    • om polecat.server start
  • Make a test change in a header and make sure it goes all the way through to the affected host(s).
  • Make a test change via rfe lcfg/ and make sure it goes all the way through to the affected host.
  • When everything seems OK, edit live/lcfg-subversion-authz.h to undo the access restrictions imposed earlier.
  • Restart rfe: om canopus.rfe start
  • Announce the all clear.
  • Re-enable subversion dumps on canopus: rfe lcfg/canopus and remove the cron.objects overrides.
  • Enable mirror backups on canopus: rfe lcfg/canopus
  • Remove the lcfg-master.h header from steen: rfe lcfg/steen
  • Re-enable rfe on steen: om steen.rfe start
  • Mail lcfg-discuss to announce the completion of the move.
  • Mail cos to announce the completion of the move.
  • Topic revision: r13 - 30 Jul 2019 - 15:05:07 - StephenQuinney
     
    This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
    Ideas, requests, problems regarding TWiki? Send feedback
    This Wiki uses Cookies