wiki.inf Pandemic docs

Basics

At the time of writing wiki.inf.ed.ac.uk is running on a VM called buckrogers. Most of the configuration comes from the live/inf-wiki.h header.

On the VM, the two interesting file locations are:

  • /disk/data/wiki/conf/ - some apache config
  • /disk/data/wiki/twiki/ - the docroot of the website

Apache is controlled, and partially configured, via the apacheconf component. The usual apache config is in /etc/httpd/, but that includes the conf file wiki.conf from the config dir above.

The site is HTTPS only, with HTTP requests redirecting to the HTTPS site. However only restricted pages/wikis require Cosign authentication.

What goes wrong?

Pretty much nothing. It just keeps going.

We used to have the occasional problem with partitions filling up, and stopping authentication working, but since it's last upgrade and repartitioning /tmp and /var are their own partitions. But full partitions might be something to check if there are issues.

Any other problems are usually related to the underlying machine, so check connectivity, general health. Or self-inflicted problems, ie a configuration change doesn't go to plan.

During a genuine pandemic type situation, presumably there won't be much changes going on, if there have been. Check the usual LCFG logs/profile for clues.

Apache is configured to log into

  • /var/lcfg/log/apacheconf.access
  • /var/lcfg/log/apacheconf.error
  • /var/log/httpd/modsec_audit.log

TWiki logs into /disk/data/wiki/twiki/data

  • log<year><month>.txt
  • warn<year><month>.txt
  • debug.txt - manually rotated very occasionally

See TraditionalWebTroubleshooting for some generic tips, but basically try restarting the apacheconf component, and check the logs for errors.

HTTPS Certificate

One thing that might happen during a pandemic, if timing is bad, is the HTTPS certificate might expire. It is currently a Quovadis signed one, but we'll probably move to a Lets Encrypt nearer to renewal time (March 2020). You could do that, with some help from Infrastructure people, or just explain to people they will have to put up the warnings from their browser.

Mirror

There is a readonly mirror of the wiki service at ikiw.inf.ed.ac.uk. It shares a lot of the configuration from the live site, so if there is a configuration problem with the live site, it may well affect the mirror. It doesn't use Cosign, and there's no authentication, so only public content on wiki.inf will me accessible on ikiw.inf.

But if the problem is with the wiki.inf VM, and you can't get it going, ikiw.inf may be useful.

Disaster recovery

Assuming that the machine hosting wiki.inf has died in an irretrievable way. Then the simplest thing is to re-create the VM with the same profile on the same network. Perhaps comment out the #include <live/inf-wiki.h> line until you've got it installed and the data restored.

You'll need to restore the data (/disk/data/wiki/) from some backup. It is mirrored nightly via the usual mirror mechanism, which then goes to tape. Or 3 times a day, just the twiki data, is mirrored to ikiw.inf, so that will contain the most recent Wiki page changes, but doesn't include the apache config, so you may want a blend of the two mirrors.

You'll also need to recover the SSL certificates, that will have been lost from /etc/httpd/conf/ssl.*/ when the original machine died.

  SSLCertificateFile "/etc/httpd/conf/ssl.crt/quovadis-wiki.inf.ed.ac.uk.crt"
  SSLCertificateKeyFile "/etc/httpd/conf/ssl.key/quovadis-wiki.inf.ed.ac.uk.key"
  SSLCertificateChainFile "/etc/httpd/conf/ssl.crt/quovadis.chain"

There should be a copy of these files in ~neilb/work/dice/certs/wiki.inf/ or Infrastructure Unit may also have copies, or can help in generating replacements.

Once the data has been restored and the certificates in place, then enabling the live/inf-wiki.h header (and probably a reboot) should restore the service.

Note that the wiki.inf host has two IPs. The usual host IP, and the service IP for wiki.inf. It doesn't matter if the replacement host IP has changed, but the service IP for wiki.inf.ed.ac.uk should remain the same, and the replacement machine should be listening on that IP. If you've changed the service IP then you'll need to update the DNS and profile appropriately.

#verbatim inf.ed.ac.uk wiki           300 IN A 129.215.32.14

But probably best to keep using the same service IP if you can.

-- NeilBrown - 05 Mar 2019

Topic revision: r1 - 05 Mar 2019 - 12:55:45 - NeilBrown
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies