Plone "Top Five"

Plone (or wcms.inf.ed.ac.uk and various aliases) is a Web Content Management System that we use to host various project web sites and, increasingly, more Informatics-based content. Currently hosted on cigar.inf, it is built using the Zope web application server written in Python. The hardware is physically located in the AT server room.

We also run a Plone instance on www.inf.ed.ac.uk these days, and it provides the /student-services/ and /school-services/ content. What follows will mostly apply to what's running on www.inf too, but paths will be different eg /disk/data/plone/zope/ for the instance location.

Although Zope provides its own HTTP server, Zserver (so Apache isn't actually needed), we do put Apache in front of the Zserver to give us Cosign authentication and other functionality. The Zserver running on cigar.inf only listens to port 8080 on the loopback device (127.0.0.1), so browsing to http://cigar.inf.ed.ac.uk:8080/ shouldn't (and better not!) work.

Apache handles the various requests for the virtual hosts, eg www.emime.org, www.not-a-service.inf.ed.ac.uk, etc, and proxies the request to localhost:8080/.... It also passes authentication information (which is taken as gospel by the Zserver), which is why you don't want the outside world to be able to talk directly to the Zserver on port 8080.

If you do need to talk directly to the Zserver, then you can log into cigar.inf and run firefox locally to talk to localhost:8080. I do

 firefox --no-remote localhost:8080 

Or use some tunneling or SOCKS proxying.

Things of Note

There is not a zope or plone component. Instead zope is started via the /etc/init.d/plone script. Which is added to boot.services. The file compoment is used to populate /etc/sysconfig/plone with the location of the instance by setting the INSTANCE variable.

INSTANCE is set to /opt/wcms/zope/ which is actually a symlink to /disk/cigar1/wcms/zope/ (this is all the data and config for the zope part of the service). It is part of /disk/cigar1/wcms/ and this is what needs to be restored following a disaster).

The Zope and Plone software ships in a single, locally built, plone RPM.

You can't just upgrade the Plone RPM and expect things to keep working - you need to go through additional steps, mainly the "portal_migration" tool for each site in the ZMI. If you must do this, try it on a clone of the wcms server first.

Cigar has multiple virtual network interfaces defined for all the HTTPS certificates we require for the cosign authenticated sites. These are on the etherbond interface bond0.

Apache is configured using the apacheconf component, which expects the included apache config files to live in /etc/httpd/conf.d/. We maintain our apache config files in /opt/wcms/apache/ (a symlink to /disk/cigar1/...) so it gets mirrored - which means that we have to symlink (via the file component) from files in /opt/wcms/apache/ to /etc/httpd/conf.d/.

How it works

See ServicesUnitPloneWCMSManagement for a bit more detail, but simply:

• a new Plone site is created in the Zope via the Zope Management Interface (ZMI) and given an id eg "dice". You need to know the zope admin username and password.

• At this point (due to the way Apache is set up to proxy zope), the site is now live at the URL wcms.inf.ed.ac.uk/zope/dice/

• If you want a friendlier URL, eg
  • wcms.inf.ed.ac.uk/dice/, then you need to update the two wcms.conf and wcms80.conf apache config files with the required Proxy and potential access restrictions
  • or something like www.dice.org or www.dice.inf.ed.ac.uk needs more work:
    • Create another new IP for cigar (needed for X509 HTTPS cert)
    • create the new X509 certificate.
    • create a new virtualhost (via LCFG, using the macros, as we are using apacheconf) and the corresponding config file(s) with the necessary Proxy and access restrictions.
• If you do want to restrict access to certain groups of people (even before zope/plone does its own authorisation) then you can do this using various apache techniques, the most common being the require group ... where the group is defined in the apache/htgroups file.

Apache is partly configure via the apacheconf component and partly by the config files in /opt/wcms/apache/ (really /disk/cigar/wcms/apache).

Common Problems

HTTPS funnies

Though we used to have this problem regularily, we are now running with threads turned off in the Zserver, which seems to have solved the problem. What happened was that the Zserver would serve the wrong content, eg the browser would ask for an image and it would get a style sheet instead, or any other odd combination. Obviously this would cause havoc with a web page. Unforuntately once a browser has what it thinks is an image for example, it will continue to use its own wrong cached copy, even though a new request to the Zserver would probably now give the correct content. So a "shift reload" is sometimes all it needs to fix a corrupt render of the web page.

Alternatively have them try accessing the site via an alternative wcms.inf.ed.ac.uk/zope/<site-id> URL. Note this will only work for DICE users, not iFriends, and may have further restrictions as defined in the apache config.

We have yet to have any reports of this mis-caching on HTTP sites.

Sites just not there

This has only happened if the network is down (someone else's problem) or if apacheconf hasn't restarted following the Sunday morning logrotate restart (very rare). However if networking is fine, then check if:

• apache is running - grep for httpd processes on cigar
• Zserver is running - grep for zope owned proceses on cigar, you

root     26583     1  0 Sep12 ?        00:00:07 /usr/lib64/plone/Python-2.4/bin/python /usr/lib64/plone/Zope-2.10.7-final-py2.4/lib/python/zdaemon/zdrun.py -S /usr/lib64/plone/Zope-2.10.7-final-py2.4/lib/python/Zope2/Startup/zopeschema.xml -b 10 -d -s /disk/cigar1/wcms/zope/var/instance/zopectlsock -x 0,2 -z /disk/cigar1/wcms/zope/parts/instance /disk/cigar1/wcms/zope/parts/instance/bin/runzope
zope     26584 26583  7 Sep12 ?        1-06:13:35 /usr/lib64/plone/Python-2.4/bin/python /usr/lib64/plone/Zope-2.10.7-final-py2.4/lib/python/Zope2/Startup/run.py -C /disk/cigar1/wcms/zope/parts/instance/etc/zope.conf

You can just try restarting apache if it isn't running, using the component. If it doesn't start, try looking in the /var/lcfg/log/apacheconf and apacheconf.error files for clues.

You may also want to check the memory footprint of zope via "top", eg

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
26584 zope      20   0 2702m 2.2g 3440 S  0.0 57.3   1813:55 python             

There appears to be a memory leak in the current zope, and it can get pretty large, so restarting it can free up lost resources.

If you want to try restarting zope, ideally do it via the zope interface at http://localhost:8080/manage, then click on "Control_Panel in the upper left, and then click the "Restart" button on the main page. This is quicker, but still leaves the site inaccessible for a while.

Alternatively stop and start it via the init.d script, though this takes longer to restart the service.

  /etc/init.d/plone stop
  /etc/init.d/plone start

Zope takes a while to start up (this can be 2-3 mins), so keep an eye on the log file /opt/wcms/zope/var/log/instance.log. When it is ready it will say "INFO Zope Ready to handle requests", or give error messages saying why it didn't start. Good luck if that's the case! Though the only time I've seen that happen is when there's been a change to the ../zope/products/ directory, eg something got deleted or there's been a major version change of something. Try reverting back to the last known good state from the backups/mirrors.

I should also say the the Zserver logs its HTTP requests like apache does, but to the file /opt/wcms/zope/var/log/instance-Z2.log. Remember that /opt/wcms/zope/ is a symlink to /disk/cigar1/zope/, which does sometimes lead to confusing warnings about "duplicate products detected" from the alternative path.

Disaster recovery

Assuming it has all gone bang, and you need to rebuild the service, then life will be a lot simpler if the new hardware re-uses all the same DNS names and IP addresses as the current cigar.inf, otherwise you'll have to track down all the sites that have non-wcms.inf.ed.ac.uk URLs, eg www.emime.org, www.classic-project.org, and update the DNS to point at the new name/IP addresses.

The service isn't completely wrapped-up in dice/options/plone-wcms-server.h, so you will also have to look at the current cigar profile and make some appropriate changes to fstab, various virtual network interfaces and #defines.

Assuming that you have now re-installed cigar, or have an equivalent machine, you should have a vanilla zope instance and a broken apache - as the data/config that is the service needs to be restored from the mirror.

(For more detail, see ServicesUnitPloneWCMSManagement#Backup_and_Restore_Disaster_Reco)

Note there is also a comment from the Infrastructure unit about the manually-installed SSL certificates: http://www.dice.inf.ed.ac.uk/units/infrastructure/Documentation/cybertrust-certificates.html

That's it.

-- NeilBrown - 30 Sep 2013