MPU Meeting Tuesday 3rd November 2015


Alastair has created an ordersync Perl module for syncing orders files into the inventory DB. There is now also a test suite of orders data. This can be used to test populating an empty DB and also changing existing data. This should allow the checking of all expected transitions.

Alastair has added code to convert the data stored in the inventory DB back into orders files. The original import process is lossy so this does not produce precisely the original files.

Still need to finish off automating the test framework. Documentation is also required.

LCFG Client Refactoring

The resourcelib and profilelib packages have been merged. It really makes sense to merge the packagelib as well since a change in one tends to result in a change in the others. This would also help porting since the order in which the libraries need to be built is written into the top-level CMake file.

To save memory the resource types are now stored as int and there are new "to" and "from" functions to convert between the string and int representations.

The various functions for setting resource attributes now check the validity of the new value and return a bool to indicate success. For the value this checking is done using the resource type specified.

Support for all the server-style mutation functions has been added. This is particularly handy when working with tag lists. At some future point we could consider converting the server over to using this library for processing source profiles.

There is now an LCFG::Resource perl module which provides an interface to the C API for LCFG resources.

All the Informatics LCFG profiles (approx. 1500) have been parsed with a C-level parser and it did not crash. A number of genuine problems were found in the XML profiles which the server had not found. LCFG users will need to be warned about the increased strictness of the new client.

Lots of work has been done on improving the feedback to the user when a problem occurs. This is mostly related to using the derivation information to help the user find the problem in the original sources whenever possible.

A problem was fixed in the way the Perl DB_File module was being used which meant the dbm file was not being released from memory.

Despite all efforts to reduce the memory consumption it is clear that some drastic changes are going to be required in the Perl API. In particular the idea of stashing all possible resource and package states for all contexts is just too expensive. It adds an overhead of 1KB per resource even when contexts are not being used. There are approximately 9000 resources in a standard DICE profile which means an additional 9MB, furthermore we want two copies of the profile in memory so that requires an extra 18MB for little benefit. Given the parsing is very fast the plan now is to do what the current client does and reparse the profile whenever contexts change (which is a rare thing) and evaluate resources with contexts as they are encountered. This has the added benefit of making the various Perl modules much simpler and easier to comprehend.

SL7 Server Base

The live wire headers have been modified so that SL7 servers have fallback nameservers. This is not a permanent solution but means they remain usable when the dns component fails to start.

The plan for "localhome" support is to use the PAM mkhomedir module instead of the localhome component.

There are issues with fibre channel support. The PV IDs are changing on reboot when using LVM. We should check that the same happens with just ext4.

We also need to check that the multipath nagios monitoring works correctly. Does it notice if a cable is unplugged?

The hwmon component has been tested with megaraid SAS. Still need to test H200 and HP raid. Could we use otaka which is a spare DL180 from the inf unit?

Miscellaneous Development

We should all blog more about our projects.

Disk encryption
Stephen has now tested installing a machine using the new encryption support. The fstab and hackparts changes should be shipped as soon as possible. Still need to finish off the work on converting existing SL7 machines. Also need to think about how to handle the encryption of the scratch disk.

The top-level page for pkgsearch has been replaced with a pointer to the relevant page.

dns component
Alastair has reviewed George's proposed changes to the dns component for SL7, they look reasonable.


The new disk for vermelha has been installed. As the old disk is "failing" rather than "failed" we ought to wipe it before sending it back. How would we handle the replacement of disks from machines which hold important data? Should some machines have a contract which allows us to destroy failed disks rather than swap them? This seems like a good reason for using full disk encryption on servers.

Spending meeting
We will have a separate meeting to discuss MPU spending plans. This will be at 9:30am on Wednesday before the Development Meeting.

Accessibility warnings
Chris has fixed the warning about the accessibility bus on SL7 by setting the NO_AT_BRIDGE environment variable using bashdefenv. See his blog for details.

openafs 1.6.15
The openafs security update is available, servers will be upgraded after the next stable release.

drupal upgrade
There was a security update for drupal.

This week

  • Alastair
    • Inventory project
      • continue working through TartarusWorkFlow
      • consider what next can be integrated into existing system, if anything
      • Write something to parse the JSON clientreport blobs and look for any major problems.
      • Document clientreport
      • Finish off order sync code
        • documentation
        • automate test process
      • Create blog article(s)
    • @home - look at using rsync from site.pkgs instead of mirroring from upstream
    • Remove default pool if ops meeting agrees
    • Experiment with different window managers under VNC (making the assumption that performance under NX will be similar)
    • Think of a use for 'atom'
    • Understand how NetworkManager works wrt init scripts
    • Deploy encrypted /tmp and swap
      • ship new fstab and hackparts ASAP (to hit stable of 11th Nov)
      • Continue work on script to modify existing machines
        • modify to be an installroot script
        • modify to wipe swap and /tmp
    • Look at RT tickets to close
    • SL7 base server
      • localhome - mark as deferred until we have /home as symlink again (and then perhaps use pam_mkhomedir)
      • check metropolitan USB and CD
      • Continue work with FC and LVM
        • try with just bog standard ext4 partition
        • check nagios notices if FC cable removed
    • Look at Reminders

  • Chris
    • Inventory project
      • continue working through TartarusWorkFlow
      • Look at clientreport modules for replacing firmwarereport
    • pkgsearch for SL7
      • reimplement as a yum web front end (yum search for keyword produce an html file of links to cgi to do yum info)
      • Need support multiple platforms
    • Liaise with George over iDRAC documentation
    • SL7 -
      • hwmon (HP raid and H200 raid)
      • Continue testing DL180 (try otaka)
      • Finish off looking at R620 (belter)
    • RT tickets close
    • Create an MPU blog
      • create a couple of SL7 server blog articles

  • Stephen
    • LCFG client refactor stage 1
      • schedule debrief meeting
    • LCFG client refactor stage 2
      • document API
      • complete combining packages
      • blog article
    • Think about PD - Interested in ZeroMQ
    • Write up how WM switchdesk mechanism works
    • RT tickets close
    • Investigate kernel component pipe moan by using shell commands instead of RPM module
    • Look at George's lcfg-dns proposal
    • Discuss reworking of wire headers with George
    • Look at bash / gnome-terminal launching wrt our X startup
    • Look at Reminders

-- AlastairScobie - 03 Nov 2015

Topic revision: r6 - 23 Sep 2019 - 13:33:38 - AlastairScobie
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies