MPU Meeting Tuesday 6th December 2016

Inventory

Alastair has been implementing more of the sub-commands of the inventory tool ii. The inventory now gets all of a machine's MAC addresses from clientreport.

Please play with the ii command and tell Alastair what you think of it.
Please also try out the REST API for the inventory system, and tell Alastair how you get on with that.

Alastair next intends to get the location functionality working again; flesh out the documentation; check the project's to do list, and incorporate virtual machines, which have so far been excluded. If the inventory is going to be the master data source for MAC addresses, it's going to have to include virtual machines. We speculated that, for instance, kvmtool create could register a null purchase order in the inventory for each new VM. This would mean that use of kvmtool would be necessary for VM creation, but that should be acceptable since we think that it already is.

LCFG Client Refactoring

Nothing happened.

MPU SL7

  • The Simple KVM service has been fully upgraded to SL7.

  • Stephen has been working on the somewhat octopodal LCFG Master upgrade.
    • The repository browsing tentacle has been upgraded - websvn proved unsuitable so viewvc will be used instead. It seemed straightforward to configure and gives a similar-looking web view of the LCFG subversion repositories.
    • Stephen noticed and fixed some problems with rfed :
      • LCFG::Authorize was failing to query a Kerberos principal for a username; he was able to fix this using Authen::Krb5.
      • We apply a local patch to GSSAPI, to copy a credentials cache. This is now possible to do using standard code.
  • BuzzSaw and Log Cabin processing now happens on the new log host copernicus (replacing the old log host tycho).

Miscellaneous Development

  • Chris has added new headers ed/options/dsu.h and dice/options/dsu.h. They will be used to add dsu to all relevant DICE level Dell hardware headers, so that dsu will be available automatically. A monthly cron job will also be added; this will mail out a simple report listing the available firmware updates.
  • Several items arose from the LCFG Annual Review 2016 (for which minutes are now available):
    • Occasionally an RPM's filename does not conform to the name-version-release rule. For such cases pkglist-tools now generates a hard link filename which conforms to the rules, and is based on the tags in the RPM's header.
    • Repository mirroring can now be done using wget for those repositories not available by rsync.
    • Matthew Richardson noted an openssh facility which tests the validity of an openssh configuration, and produced a patch which adds this functionality to the openssh component (Bug:930).
  • Alastair has created project 407 - rewrite of lcfg-fstab component.

Operational

  • Now that the KVM service is on SL7, we need to think about its future shape. Here's a KVM miscellany:
    • While upgrading the KVM servers, Chris amalgamated multiple storage pools together, so each KVM server generally just has one storage pool of local disks.
    • With the increasing age of circle it seems increasingly dodgy to be using it as a backup server for waterloo, so we propose to move oyster to Appleton Tower to be _waterloo_'s companion KVM server. (_oyster_'s current VMs will first be moved to other Forum-based KVM servers.)
    • Moving oyster to Appleton Tower leaves less spare space on the Forum KVM servers than we would like. We propose to tackle this in several ways:
      • By replacing a number of the MPU's VMs currently in the Forum with AT-based equivalents. For instance, moving the release testing VMs there would liberate about half a terabyte on a Forum-based KVM server.
      • The IBM SAN is currently not in active use. We will create three 1TB volumes on it and add one to each of the remaining Forum-based KVM servers (azul, gaivota and girassol) for use as storage pools for KVM. The IBM-based storage pools will be used only for emergencies and as temporary decant space. The Forum-based KVM servers are already attached to the FC fabric.
      • With the retirement of jubilee and hammersmith we now have six spare 3.5" 600GB disks available for emergency use in the remaining KVM servers - at least those which can take 3.5" disks, namely oyster, azul, waterloo, amarela and vermelha. The disks are currently in KVM servers but unconfigured. Their locations will be documented, as will procedures for adding them to provide extra KVM space and for removing them again afterwards.
    • A new version of qemu-kvm-rhev (2.3.0-31.el7.23) is on test on circle. If all seems well it will be introduced to the other KVM servers soon.
  • There are no longer any SL6.6 DICE hosts. All SL6 DICE hosts are now running SL6.8.
  • The Scientific Linux maintainers intend to release a large batch of security updates on 14 December (see this message for details). Since these are likely to pull in new versions of major components of SL, we intend to delay introducing these to LCFG and DICE until January 2017. Exceptions will be made only for critical security fixes.
  • The final LCFG release of 2016 will come out on Wednesday 21 December, based on a snapshot taken on 19 December.
  • The first LCFG release of 2017 will come out on Wednesday 11 January. It will be based on a snapshot taken on Monday 9 January. We expect this first release of 2017 to include all of the SL security updates from 14 December (see above).
  • Stephen has been investigating a network problem with SL7 multi-homed machines. So far he has been unable to reproduce it.
  • Since the LCFG slave vermeer is now consistently about 30% slower than the other slave rembrandt, thanks apparently to the less capable CPUs of its KVM host, Chris will replace vermeer with a new and hopefully faster slave hosted on a newer KVM server.

This Week

  • Alastair
    • Inventory project
      • continue working through TartarusWorkFlow
      • Document clientreport (eg how to add modules)
      • Document order sync code
      • Document hpreport processing script
      • Continue work on RESTful API - TartarusRESTAPI
      • Document REST API
      • Further encourage people to use API and ii commands
      • Write more of the ii commands and document as writing.
      • Speak to George about macaddr/space feed
      • Start work on final report!
      • Chase Tim about theon acccess credential for feed
      • Convert from mod-auth_kerb to mod-auth_gssapi (See Stephen for details)
      • How represent VMs
    • Deploy encrypted /tmp and swap conversion script
      • Deploy as soon as possible
      • Need to warn users that Gnome3 may pop up a window about /tmp being full (when script is run)
    • Schedule MPU meeting to discuss systemd ordering
    • submit polkit bug to redhat - with Stephen
    • MPU SL7
      • Chase Toby again about testing latest perl-Moose under prometheus (and then make live) after October 1
        • Toby reckons now fine - will update immediately after Xmas
      • Upgrade computing.help servers
        • backup and staging servers (hjaelpe and brent) replaced by SL7 equivalents (hulp and ayuda)
        • Old servers (and nerano prototype) need removed
      • Consider whether ordershost could move to bandama
        • waiting on Stephen finishing rfe support for SL7
    • Check sysmans (et al) have 'nograce'.
    • Take a look at RT #78875
    • With Chris, setup IBM array volumes for Forum based KVM servers
      • Volumes created on IBM array. Need to configure on KBM servers (azul, gaivota, girassol)
        • girassol done
        • gaivota refusing to 'pvcreate' the new volume - claims not found or filtered, but can't see obvious reason why
        • azul - PV/VG wasn't re-created for SL7, so not using the new UUID scheme. As a result, the component isn't running properly as it can't find the PV (see blog item). Because the component isn't running, we can't create a VG on the new FC volume (using the component).
    • Do drupal update for computing.help etc

  • Chris
    • Inventory project
      • Continue work on clientreport modules for replacing firmwarereport
      • Try REST API
    • pkgsearch for SL7 -> activities list
      • reimplement as a yum web front end (yum search for keyword produce an html file of links to cgi to do yum info)
      • Need support multiple platforms
    • MPU SL7
      • wake.inf.ed.ac.uk
    • Roll out fixed sleep code
    • Any remaining work with deploying 'dsu'
      • Set up script to mail out monthly firmware update reminders (from ops meeting)
      • adding to dice hw headers
    • Reschedule MPU futures meeting
    • With Alastair, setup IBM array volumes for Forum based KVM servers
    • Replace waterloo lcfg slave with one on one of the KB KVM servers

  • Stephen
    • Inventory project
      • Try REST api
    • LCFG client refactor stage 1
      • schedule debrief meeting
    • LCFG client refactor stage 2
      • testing and documentation
      • blog article (once documentation complete)
    • Investigate kernel component pipe moan by using shell commands instead of RPM module => waiting on 7.2 => activities list
    • LCFG server symlink to exam branches - produce reporting script and discuss with Graham
    • Circulate dmesg proposal -> activities list
    • submit polkit bug to redhat - with Alastair (check under 7.3)
    • SL7 MPU
      • Continue with LCFG master server
      • Schedule juice upgrade
    • Investigate George's multiple network interfaces SL7 issue (eg consoles server)
      • waiting on George breaking metropolitan
    • LCFG annual review - produce minutes

-- AlastairScobie - 06 Dec 2016

Topic revision: r13 - 24 Sep 2019 - 13:50:24 - AlastairScobie
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies