LCFG Service Information

The LCFG service is a server-client model with a single master and multiple slaves. We normally run two main slaves, a DIY DICE slave and one or more test servers.

This page is not an exhaustive explanation of how the entire service operates or how it is configured, rather it is intended as a rough guide on where to look for more information, where configuration data should be stored and how to carry-out standard tasks.

TODO: Add basic architecture docs

Master Server

The LCFG master is currently steen. This should be referred to as lcfg-master in any configuration.

The LCFG master hosts rfe and subversion (webdav and websvn) repositories for the LCFG config data and source code. It also holds the master copy of the LCFG component schemas (default files), the stable/testing releases and the Informatics inventory. Basically any data needed by an LCFG slave should come from the master server via rsync.

All the configuration for this service is done via the following headers:

  • dice/options/lcfg-master-server.h
  • live/lcfg-master-server.h
  • live/lcfg-slave-servers-list.h
  • live/lcfg-subversion-authz.h

All the configuration should go into the release-controlled dice/options/lcfg-master-server.h (or the relevant header included from within that header).

The live/lcfg-master-server.h is intended to be purely for emergency overrides of normal settings and the introduction of new features and configuration settings when it is necessary to avoid the standard wait of a week or so for the settings to get through the release cycle.

live/lcfg-slave-servers-list.h contains lists of servers which need access to the data stored on the master. As well as the slave servers this includes the LCFG webserver and nagios monitoring servers. Host names should not be hardwired anywhere else in the configuration, use macros so they can be changed immediately and easily where necessary.

live/lcfg-subversion-authz.h is used to control access to the subversion repositories for the LCFG configuration data and source code. Most of the ACLs are done via groups to simplify management. Projects where external users need write-access need a bit more effort, there are macros to aid the process.

Disaster Recovery

There is information available on how to rebuild the LCFG master. There is also information on how to recover the LCFG service from various disaster situations.

Extra svn ACLs

By default all LCFG source code is fully accessible by all Informatics COs and CSOs, other users just have read-access. If it is necessary to give write-access to external users they need to be explicitly defined for each project.

Where possible use groups as this can vastly reduce the number of resources which need to be manually specified for the ACLs.

An example is:

_SVN_SOURCE_PROJECT(source,lcfgcups,lcfg-cups)
_SVN_SOURCE_PROJECT_WRITEACCESS(source,lcfgcups,is,is)

The first macro takes the svn repository name, an LCFG-compatible tag name and the real name of the project. This will create a separate entry in the svn authorization file but will not do anything else to alter the ACLs. The second macro takes the name of the svn repository, the LCFG-compatible tagname for the project, an LCFG-compatible tag used to identify the user or group and the real name of the user or group.

To add access for an individual user it would be something like this:

_SVN_SOURCE_PROJECT(source,lcfgcups,lcfg-cups)
_SVN_SOURCE_PROJECT_WRITEACCESS(source,lcfgcups,kenny,kenny@EASE.ED.AC.UK)

NOTE that this merely adjusts the permissions for a subversion directory. It doesn't actually create the directory itself. To do that, use a command such as:
svn mkdir https://svn.lcfg.org/svn/source/{tags,trunk,branches}/lcfg-foobar -m "Created"

Ideally you should also add a component in the LCFG bug tracker if one does not already exist and set the default assignee to something sensible.

For external write-access it is also necessary to create the relevant branches and tags directories as we do not give out global access to the necessary directories. As yourself:

$ mkdir /tmp/lcfg-source
$ cd /tmp/lcfg-source
$ svn checkout --non-recursive https://svn.lcfg.org/svn/source/branches
$ cd branches
$ svn mkdir lcfg-foo
$ svn commit -m "Added branches directory for lcfg-foo"
$ cd /tmp/lcfg-source
$ svn checkout --non-recursive https://svn.lcfg.org/svn/source/tags
$ svn mkdir lcfg-foo
$ svn commit -m "Added tags directory for lcfg-foo"

Managing the Main LCFG Slave Servers

Informatics has two main LCFG slave servers, lcfg1 (vega) and lcfg2 (altair)

lcfg1 is also accessible via the standard lcfg and lcfghost aliases. If the server with these aliases is not available then installs will not work. To avoid upsetting COs it is a good idea to update the DNS well in advance of planned downtime for that server (or do it out-of-hours).

We have two instead of one because:

  • if one breaks, we'll hopefully still have a working one
  • back when we had one LCFG server, we found that a lot of the load on it came from Apache. With the two-servers solution, the servers duplicate the LCFG profile building, but split the Apache load between them. The clients know that both servers are offering the same new profile, and tend to pick one server at random to get it from.

Configuration is done via these files:

  • dice/options/lcfg-slave-server.h
  • live/lcfg-slave-server.h

All the configuration should go into the release-controlled dice/options/lcfg-slave-server.h (or the relevant header included from within that header).

The live/lcfg-slave-server.h is intended to be purely for emergency overrides of normal settings and the introduction of new features and configuration settings when it is necessary to avoid the standard wait of a week or so for the settings to get through the release cycle.

When a new slave server is added it MUST be added to the relevant macro in live/lcfg-slave-servers-list.h so that it has rsync access to the LCFG master.

As well as the two main slave servers there are usually several test LCFG servers.

The bits of config not held in the header file are:

  • Every server MUST have the LCFG_SERVICE_ALIAS macro defined. This is the short name (e.g. lcfg2 or lcfg4) not the FQDN. The LCFG_SERVICE_ALIAS cpp variable has to be defined before the inclusion of the lcfg-slave-server.h header file.

  • One server (and only one) should be the LCFG cron server, this is set by defining the LCFG_SLAVESERVER_CRON before including dice/options/lcfg-slave-server.h in the LCFG profile.

Disaster Recovery

If an LCFG slave server dies then don't bother trying to resurrect, just do a new install, there is no local data which needs to be preserved. Once the replacement server is active and has finished the full profile build you will need to clear the cache on the other server and restart the lcfg server to avoid them being too far out of sync.

-- StephenQuinney - 24 Jan 2019

Topic revision: r12 - 24 Jan 2019 - 14:44:34 - StephenQuinney
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies