Moving the LCFG Master to a new machine

This is how we plan to move the LCFG master from schiff to steen.

Beforehand

Every day or so until the day of the move:
  • Copy over the rfe data from schiff to steen:
    • rsync -av --delete schiff::rfedata/ /var/rfedata/
  • Copy over the dumps of the subversion repositories:
    • rsync -av --delete schiff::lcfgsvn/ /var/lcfg/svndump/
  • Copy /var/svn:
    • rsync -av  --delete schiff::svndatadir/ /var/svn/
  • Copy over the testing and stable releases:
    • rsync -av --delete schiff::lcfgtestingrelease /var/lcfg/releases/testing/
    • rsync -av --delete schiff::lcfgstablerelease /var/lcfg/releases/stable/

On the day

  • Mail lcfg-discuss to announce the start of the move.
  • Mail cos to announce the start of the move.
  • Pacify Nagios for schiff and steen, vermeer and rembrandt, vole, sauce, sernabatim and barents
  • Lots of windows on schiff and steen before we start. nsu in some of them.
  • Change the permissions on the subversion repositories (in live/lcfg-subversion-authz.h) so that only MPU members have rw permission for the lcfg subversion repositories. Test this after it's been done.
  • Stop the LCFG slave servers:
    • om vermeer.server stop
    • om rembrandt.server stop
    • om vole.server stop
    • om sauce.server stop
    • om sernabatim.server stop
    • om barents.server stop
  • Make 'rfe' read-only on both old and new servers. This could be done by stopping the rfe component:
    • om schiff.rfe stop
    • om steen.rfe stop
  • Copy over the rfe data from schiff to steen:
    • rsync -av --delete schiff::rfedata/ /var/rfedata/
  • Dump the subversion repositories on schiff - this takes a minute or two:
    • svnadmin dump /var/svn/dice > /var/lcfg/svndump/dice/dice.schiff-final.dump
    • svnadmin dump /var/svn/lcfg > /var/lcfg/svndump/lcfg/lcfg.schiff-final.dump
    • svnadmin dump /var/svn/source > /var/lcfg/svndump/source/source.schiff-final.dump
  • Copy over the dumps of the subversion repositories: rsync -av --delete schiff::lcfgsvn/ /var/lcfg/svndump/
  • Load the repository data into the new repositories. The proper way to do this would be to run svnadmin load into fresh empty repositories from each schiff-final.dump file. However that would take all day, so instead we just use rsync to copy the whole of /var/svn at once:
    • rsync -av  --delete schiff::svndatadir/ /var/svn/
  • For reference this is what the svnadmin load commands would be if we were using them:
    • svnadmin load /var/svn/dice < /var/lcfg/svndump/dice/dice.schiff-final.dump
    • svnadmin load /var/svn/lcfg < /var/lcfg/svndump/lcfg/lcfg.schiff-final.dump
    • svnadmin load /var/svn/source < /var/lcfg/svndump/source/source.schiff-final.dump
  • Copy over the testing and stable releases:
    • rsync -av --delete schiff::lcfgtestingrelease /var/lcfg/releases/testing/
    • rsync -av --delete schiff::lcfgstablerelease /var/lcfg/releases/stable/
  • Move lcfg-master and lcfgsvn from schiff to steen: rfe dns/inf
  • Move svn.lcfg.org from schiff to steen: rfe dns/lcfg_org
  • Update the DNS on each LCFG server:
    • om schiff.dns update
    • om steen.dns update
    • om vermeer.dns update
    • om rembrandt.dns update
    • om vole.dns update
    • om sauce.dns update
    • om sernabatim.dns update
    • om barents.dns update
  • Make a change to a header to trigger the first autocheckout on the new server. See if something appears in /var/lib/autocheckout/lcfg. (Alternatively it could be rsynced from the old machine?)
  • Restart the test slave:
    • om vole.server start
  • If all is OK, restart diydice:
    • om sernabatim.server start
  • If all is OK, restart the main LCFG slave servers:
    • om vermeer.server start
    • om rembrandt.server start
  • and the others:
    • om sauce.server start
    • om barents.server start
  • Make a test change in a header and make sure it goes all the way through to the affected host(s).
  • Make a test change via rfe lcfg/ and make sure it goes all the way through to the affected host.
  • When everything seems OK, edit live/lcfg-subversion-authz.h to undo the access restrictions imposed earlier.
  • Restart rfe: om steen.rfe start
  • Announce the all clear.
  • Re-enable subversion dumps on steen: rfe lcfg/steen and remove the cron.objects overrides.
  • Enable mirror backups on steen: rfe lcfg/steen
  • Remove the lcfg-master.h header from schiff: rfe lcfg/schiff
  • Re-enable rfe on schiff: om schiff.rfe start
  • Mail lcfg-discuss to announce the completion of the move.
  • Mail cos to announce the completion of the move.
  • Ordershost

    Ordershost does not need to be on the same machine as lcfg-master, but we do want it to end up on steen eventually. We intend to leave it on schiff until after lcfg-master has moved to steen, then move it to steen separately.
    Edit | Attach | Print version | History: r13 | r11 < r10 < r9 < r8 | Backlinks | Raw View | Raw edit | More topic actions...
    Topic revision: r9 - 30 Jun 2014 - 13:51:10 - ChrisCooke
     
    This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
    Ideas, requests, problems regarding TWiki? Send feedback
    This Wiki uses Cookies