MPU Meeting Thursday 24th November 2016

Inventory

Alastair has added POD documentation for the various ii commands. He has been adding more commands (and documenting them as he goes along). He has also rewritten a number of existing commands to use the REST API.

LCFG Client Refactoring

Nothing happened.

MPU SL7

KVM Service
The two KVM servers in AT - circle and waterloo have now been upgraded to SL7. There was a problem with missing storage pools on oyster which has now been fixed. Only the KB servers are now left to be upgraded, they will be done soon. The additional disks which have been swapped around are clearly labelled so we know their heritage.

LCFG web site
The LCFG website upgrade just worked. The wiki was upgraded to TWiki version 6 which took a bit more effort, it looks a bit prettier than the previous version.

openafs buildhost
The only problem with this upgrade was that budapest is an ancient Dell PowerEdge 1950 with an old SMBIOS version. This means that the network interface names enp5s0 and enp9s0 rather than the expected standard enoX style. It might be possible to upgrade the BIOS but given this machine is old and will be retired soon we don't really care that much.

package master
The upgrade of the package master was very straightforward, it just required a few tweaks to the apache configuration to support 2.4.

LCFG slaves
Whilst upgrading the LCFG slaves Stephen spotted that they didn't have sufficient memory to cache all the source files and generated XML files. The memory has been increased from 2GB to 8GB which has produced clear performance benefits. The times for a rebuild of 960 profiles were: rembrandt 32 mins (on gaivota) and vermeer 41 mins (on waterloo). It appears that on average vermeer is about 30% slower to process a profile. We could replace vermeer with an LCFG slave on a newer KVM server, maybe at KB once those KVM servers have been upgraded to SL7?

LCFG master
This upgrade is going to take a lot of effort as there are many different aspects. Alastair has agreed to help simplify the process by moving the ordershost to a different machine (possibly bandama). It looks like the apache mod_gssapi can be used to replace mod_krb5, Stephen will document the config necessary. Stephen had a battle with the locally-written mod_user_rewrite which needed to be ported to the "new" LCFG build tools, thankfully it still works with apache 2.4. Stephen has reworked the rfe package so that there is a separate rfe-server sub-package, this has also gained systemd support. The rfe LCFG component has been completely rewritten in Perl and has a Perl TT template which is a lot nicer. For subversion we don't need xinetd support so that has been made an option which has to be deliberately enabled for SL7. websvn will be replaced with viewvc, Stephen is working on a component which can manage the configuration. Unlike websvn it's completely configurable using a single text file which is a definite bonus. Lots of LCFG defaults packages have been updated but there are still over 50 to do, it will be a slow process, Stephen will do some more once the changes have made it into the next stable release.

Miscellaneous Development

Software Collections
At short notice Scientific Linux decided to drop all support for Software Collections. This meant we had to switch to the repository provided by Centos. This required updates to nearly all SL6 collections as we were a long way behind on software versions and they no longer existed in the Centos repository.

yum component
Chris has added support for excluding packages from yum repository configurations, he will roll out the changes this week.

Operational

KB UPS
Chris has added checks for the KB UPS to the MPU servers. This should give us 5 minutes notice of any power outage, this might not be enough for the KVM servers but is better than nothing.

Dell R220
As there only a small number of these servers in use by the Inf Unit we had not previously checked the SL7 support for the Dell R220. This meant that Toby had problems with network configuration. It also has some weird problem with serial console support where the grub menu is apparently invisible. Changing the "Redirection after Boot" BIOS option seems to have fixed it, we need to remember why we used to always disable that option.

krb5 packages
Toby has been building newer versions of the krb5 packages for the KDCs. This caused problems for pkgforge builds with the "world" bucket and then later on the "inf" bucket. Stephen fixed this by configuring pkgforge to ignore local builds of the krb5 packages. Toby can still build the packages using mock on his machine by removing those exclusions from the config.

SL6.6
The remaining LCFG profiles using the DICE_STICK_WITH_SL66 macro will be modified next Monday so that all SL6 machines are running on SL6.8.

This Week

  • Alastair
    • Inventory project
      • continue working through TartarusWorkFlow
      • Document clientreport (eg how to add modules)
      • Document order sync code
      • Document hpreport processing script
      • Continue work on RESTful API - TartarusRESTAPI
      • Document REST API
      • Further encourage people to use API and ii commands
      • Write more of the ii commands and document as writing.
      • Speak to George about macaddr/space feed
      • Start work on final report!
      • Chase Tim about theon acccess credential for feed
      • Convert from mod-auth_kerb to mod-auth_gssapi (See Stephen for details)
    • Remove default pool if ops meeting agrees
    • Deploy encrypted /tmp and swap conversion script
      • Deploy as soon as possible
      • Need to warn users that Gnome3 may pop up a window about /tmp being full (when script is run)
    • Schedule MPU meeting to discuss systemd ordering
    • package up ILW stuff and document process
    • submit polkit bug to redhat - with Stephen
    • MPU SL7
      • Chase Toby again about testing latest perl-Moose under prometheus (and then make live) after October 1
      • Upgrade computing.help servers
    • Check sysmans (et al) have 'nograce'.
    • Take a look at RT #78875
    • Tidy up IBM array again (remove jubilee and hammersmith)
    • Consider whether ordershost could move to bandama
    • Check maipo RH7.3 (and other RHel box)

  • Chris
    • Inventory project
      • Continue work on clientreport modules for replacing firmwarereport
      • Try REST API
    • pkgsearch for SL7
      • reimplement as a yum web front end (yum search for keyword produce an html file of links to cgi to do yum info)
      • Need support multiple platforms
    • MPU SL7
      • Continue with KVM server upgrades (KB)
    • Roll out fixed sleep code
    • Any remaining work with deploying 'dsu'
    • Consider spending plan for 2017 onwards
    • Reschedule MPU futures meeting
    • Submit SL7 server base project to development meeting
    • Set up script to mail out monthly firmware update reminders (from ops meeting)

  • Stephen
    • Inventory project
      • Try REST api
    • LCFG client refactor stage 1
      • schedule debrief meeting
    • LCFG client refactor stage 2
      • testing and documentation
      • blog article (once documentation complete)
    • Investigate kernel component pipe moan by using shell commands instead of RPM module => waiting on 7.2 => activities list
    • LCFG server symlink to exam branches - produce reporting script and discuss with Graham
    • Circulate dmesg proposal -> activities list
    • submit polkit bug to redhat - with Alastair (check under 7.3)
    • SL7 MPU
      • continue work with buzzsaw
      • Continue with LCFG master server
    • Check hardware model headers to make sure all models support new network naming scheme for SL7
    • Consider spending plan for 2017 onwards
    • Investigate George's multiple network interfaces SL7 issue (eg consoles server)
      • try physical machine with real interfaces first
    • LCFG annual review

-- AlastairScobie - 24 Nov 2016

Topic revision: r8 - 24 Sep 2019 - 13:50:24 - AlastairScobie
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies