Updating LCFG to a new minor release

The generation of package lists is done using the lcfg-yummy tool, there are some scripts in the pkglist-tools package which will do most of the work. For SL6 any x86_64 machine will do, it can be SL6 or SL7 (e.g. your own desktop machine), just add the following to the LCFG profile:

#include <dice/options/pkglist-tools.h>
#include <dice/options/yummy.h>
#include <live/release-scripts.h>

Package Lists

Firstly create a complete set of empty package list files for both the lcfg and dice levels, use a previous minor release as a guide to which files are required. For example, creating sl75 files based on sl74:

% cd core/packages
% for old in {dice,lcfg}/*sl74*.rpms; do \
        new=$(echo $old | sed -e 's/sl74/sl75/');\
        test -f $new || ( touch $new && svn add $new); \
done

svn commit -m "Added LCFG level package lists for SL7.5" lcfg/*_sl75_*.rpms
svn commit -m "Added DICE level package lists for SL7.5" dice/*_sl75_*.rpms

These files need to exist so that it is possible to test the various package list combinations, without them the various test LCFG profiles will not compile.

Headers

There are a couple of headers which need to be created for each minor release, these can be copied from the previous versions and modified appropriately. For example:

svn cp core/include/lcfg/hidden/sl74.h core/include/lcfg/hidden/sl75.h

There is also a live/tests-pkglist-sl7.h header which needs to be updated so that the new package lists can be tested. In the top block ensure that the two new "hidden" files are included. This is what we have for SL7.4:

#include <lcfg/hidden/sl74.h>
#include <lcfg/hidden/sl7vars.h>

Also ensure that the profile.release resource in that header has been set to develop.

The headers (lcfg, ed, dice and live) must all be searched manually to find all references to the profile.packages_options resource. Each one needs to be checked to see if the versions are still correct for the new minor update. Similarly the LCFG profiles for DICE need to be checked. There is no simple way to automate that process so it's all a bit tedious...

Generate the kernel and updates lists

The updates and kernel package lists are generated using the /usr/share/pkglist-tools/scripts/build_package_lists script which is driven by the /etc/pkglist-tools/lcfg_updates.ini configuration file. There needs to be an entry in this file for each architecture of each minor platform (e.g. sl74 and sl75). The entries for SL7 look like:

[platform sl75]
distro=sl7
releasever=7.5
arch=x86_64

[platform sl74]
distro=sl7
releasever=7.4
arch=x86_64

That file is not managed by LCFG so don't edit the file directly, change the master copy in the pkglist-tools package and create a new package. Once the new package is installed on your machine just run the script in the correct directory in your svn working copy, note that it takes a little while.

% cd svn/lcfg/core/packages/lcfg/
% /usr/share/pkglist-tools/scripts/build_package_lists

The expected output is something like this and should include your new minor platform:

Making sl68 updates package lists
Making sl68_64 updates package lists
Making sl74 updates package lists
Making sl75 updates package lists
Making SL 6.8 kernel package lists
Making SL 7.4 kernel package lists
Making SL 7.5 kernel package lists

Commit the new files into svn, e.g.:

svn commit -m "SL7.5 updates and kernel package lists" lcfg_sl75_updates.rpms lcfg_sl75_kernel.rpms

There may be updates for other platforms, they must be reviewed (check with svn diff) before committing. Note that the sorting of packages is annoyingly unstable so sometimes a file changes although there are no actual functional changes.

LCFG Release package

A new minor release needs to be made of the relevant lcfg-release package (e.g. =lcfg-release-el7):

cd svn/source/trunk/lcfg-release-el7/
lcfg-reltool minor

For SL7.5 this gives the following output:

No version-control information in the LCFG metafile for this project.
Auto-detected that the SVN module should be used.
Checking that all file changes have been committed.
Generating Changelog from subversion log
Updating build ID file
Tagging lcfg-release-el7 at version 7.5.0

Build the package and then submit the SRPM for building by pkgforge in the lcfg bucket for the correct platform:

pkgforge submit -B lcfg -p el7 /afs/inf.ed.ac.uk/user/s/squinney/lcfgbuild/lcfg-release-el7-7.5.0/lcfg-release-el7-7.5.0-1.src.rpm

Once built the package should be added to the list in the lcfg/defaults/profile.h header:

#ifdef LINUX_EL7

#if OS_RELEASE_MINOR == 3
!profile.packages      mEXTRA(lcfg-release-el7-7.3.0-1/noarch:br)
#elif OS_RELEASE_MINOR == 4
!profile.packages      mEXTRA(lcfg-release-el7-7.4.1-1/noarch:br)
#else /* 7.5 */
!profile.packages      mEXTRA(lcfg-release-el7-7.5.0-1/noarch:br)
#endif

#endif /* LINUX_EL7 */

The :br flags are essential as this is how we guarantee that a reboot will occur when a machine updates to the new minor release.

Update yummy configuration

The yummy configuration files are stored in subversion alongside the packages directory:

% svn co https://svn.lcfg.org/svn/lcfg/core
% ls core
include  packages  validate  yummy

Inside the yummy directory there are sub-directories which match with those in the packages directory.

Due to manual edits, over time the package lists tend to drift away from the yummy templates which were originally used to create the files. The first step is to ensure the yummy files are up-to-date, this can be done on any machine for all platforms and architectures.

The base and desktop package lists are architecture specific, the default architecture for the packages must be specified. Take the lists for the current minor release (e.g. SL7.4) and produce the release-independent yummy files.

for i in base desktop; do \
    /usr/share/pkglist-tools/scripts/pkglist_to_yummy --arch x86_64 packages/lcfg/lcfg_sl74_$i.rpms \
           > yummy/lcfg/lcfg_el7_$i.yummy ;\
done

It is important to note for the base lists that the kernel package will be missing from the yummy file, you will need to manually add this back to the yummy file.

The following lists are used by both architectures:

for i in installbase installroot international kernel kvm options; do \
   /usr/share/pkglist-tools/scripts/pkglist_to_yummy packages/lcfg/lcfg_sl74_$i.rpms \
           > yummy/lcfg/lcfg_el7_$i.yummy;\
done

Each of the base, installroot and installbase yummy files will need a kernel package entry adding to satisfy dependencies. They don't appear in the LCFG package lists since the kernel package list is in a separate package list which is included.

The differences in the yummy files now need to be examined (using svn diff), most of them will just be additions of extra packages. If all looks good the changes can be committed.

Package List Creation

The generation of the package lists is done in stages, each stage requires the lists generated by the previous stage to be correct.

Stage 1 : Generate the base package lists

The generation of the base, installbase and installroot package lists are done in stage 1:

/usr/share/pkglist-tools/scripts/make_sl7_stage1 7.5

Replace 7.5 with the required minor release version. If all goes well, the output from the script will look something like:

Loaded plugins: kabi, priorities
Loading support for Red Hat kernel ABI
291 packages excluded due to repository priority protections
warning: Generating 12 missing index(es), please wait...

Generated /disk/scratch/lcfgpkgs_9XCD2w/lcfg_sl75_base.rpms

Loaded plugins: kabi, priorities
Loading support for Red Hat kernel ABI
291 packages excluded due to repository priority protections

Generated /disk/scratch/lcfgpkgs_9XCD2w/lcfg_sl75_installbase.rpms

Loaded plugins: kabi, priorities
Loading support for Red Hat kernel ABI
291 packages excluded due to repository priority protections

Generated /disk/scratch/lcfgpkgs_9XCD2w/lcfg_sl75_installroot.rpms

Results are in /disk/scratch/lcfgpkgs_9XCD2w

If any of the yummy runs failed then the yummy configs will need to be editted in your svn working copy and the changes submitted. Continue to iterate until the package lists are all successfully generated.

If successful, copy the generated package lists into the appropriate location in your svn working copy and commit the changes.

For the installroot the latest kernel and linux-firmware packages will need to be added to the lcfg_el7_lcfg_installroot.rpms package list (you can get the versions from the lcfg_sl75_kernel.rpms package list that you generated earlier. The associated kernel-install-sl7 package can be built later. It should look something like this:

#else  /* 7.5 onwards */

kernel-3.10.0-862.3.2.el7
linux-firmware-20180220-62.git6d51311.el7/noarch

#endif

The new package lists must next be tested to ensure they work with updaterpms and do not conflict with other headers and package lists. Firstly check on the LCFG test server status page that the relevant LCFG profiles have compiled. The profile names have a sl6-pkglist- or sl7-pkglist- prefix, at this stage the profiles of interest are 1, 2, 3, 4, 9, 10.

If they have all compiled then run the test script for each platform (sl7) like this, as root:

/usr/lib/lcfg/release-scripts/scripts/test_package_lists --platform sl7 --tests 1,2,3,9,10

Check for any errors and modify any package lists (e.g. lcfg_el7_lcfg.rpms) and headers where necessary.

Stage 2 : Generate the desktop and options package lists

The generation of the desktop, options and kvm package lists are done in stage 2 (also kvm, scl and latex for SL7):

/usr/share/pkglist-tools/scripts/make_sl6_stage2 6.6

Replace 6.6 with the required minor release version. If all goes well, the output from the script will look something like:

The chroot for profile 'sl6-pkglist-01' is /tmp/lcfgpkgs_tweWp1/lcfgtests/sl6-pkglist-01
Loaded plugins: kabi, priorities
Loading support for Red Hat kernel ABI
380 packages excluded due to repository priority protections

Generated /tmp/lcfgpkgs_tweWp1/lcfg_sl66_desktop.rpms

The chroot for profile 'sl664-pkglist-01' is /tmp/lcfgpkgs_tweWp1/lcfgtests/sl664-pkglist-01
Loaded plugins: kabi, priorities
Loading support for Red Hat kernel ABI
487 packages excluded due to repository priority protections

Generated /tmp/lcfgpkgs_tweWp1/lcfg_sl66_64_desktop.rpms

Loaded plugins: kabi, priorities
Loading support for Red Hat kernel ABI
487 packages excluded due to repository priority protections

Generated /tmp/lcfgpkgs_tweWp1/lcfg_sl66_options.rpms

Loaded plugins: kabi, priorities
Loading support for Red Hat kernel ABI
487 packages excluded due to repository priority protections

Generated /tmp/lcfgpkgs_tweWp1/lcfg_sl66_kvm.rpms


Results are in /tmp/lcfgpkgs_tweWp1

If any of the yummy runs failed then the yummy configs will need to be editted in your svn working copy and the changes submitted. Continue to iterate until the package lists are all successfully generated.

If successful, copy the generated package lists into the appropriate location in your svn working copy and commit the changes.

The new package lists must next be tested to ensure they work with updaterpms and do not conflict with other headers and package lists. Firstly check on the LCFG test server status page that the relevant LCFG profiles have compiled. The profile names have a sl6-pkglist- or sl664-pkglist- prefix, at this stage the profiles of interest are 5, 6, 7, 8, 14.

If they have all compiled then run the test script for each platform (sl6 and sl664) like this:

/usr/lib/lcfg/release-scripts/scripts/test_package_lists --platform sl664 --tests 5,6,7,8,14
/usr/lib/lcfg/release-scripts/scripts/test_package_lists --platform sl6 --tests 5,6,7,8,14

Check for any errors and modify any package lists (e.g. lcfg_sl6_lcfg.rpms) and headers where necessary.

Stage 3 : Generate the international package lists

The generation of the international packages list is done in stage 3:

/usr/share/pkglist-tools/scripts/make_sl6_stage3 6.6

Replace 6.6 with the required minor release version. If all goes well, the output from the script will look something like:

The chroot for profile 'sl664-pkglist-05' is /tmp/lcfgpkgs_48SLZe/lcfgtests/sl664-pkglist-05
Loaded plugins: kabi, priorities
Loading support for Red Hat kernel ABI
649 packages excluded due to repository priority protections

Generated /tmp/lcfgpkgs_48SLZe/lcfg_sl66_international.rpms


Results are in /tmp/lcfgpkgs_48SLZe

If any of the yummy runs failed then the yummy configs will need to be editted in your svn working copy and the changes submitted. Continue to iterate until the package lists are all successfully generated.

If successful, copy the generated package lists into the appropriate location in your svn working copy and commit the changes.

The new package lists must next be tested to ensure they work with updaterpms and do not conflict with other headers and package lists. Firstly check on the LCFG test server status page that the relevant LCFG profiles have compiled. The profile names have a sl6-pkglist- or sl664-pkglist- prefix, at this stage the profiles of interest are 15.

If they have all compiled then run the test script for each platform (sl6 and sl664) like this:

/usr/lib/lcfg/release-scripts/scripts/test_package_lists --platform sl664 --test 15
/usr/lib/lcfg/release-scripts/scripts/test_package_lists --platform sl6 --test 15

Check for any errors and modify any package lists (e.g. lcfg_sl6_lcfg.rpms) and headers where necessary.

Installer kernel

The kernel used in the LCFG installer for the platform should be updated to the most recent available. See LCFGLinuxInstall for details.

Installbase profiles

New installbase and diceinstallbase profiles should be generated, and tested with installs. The installroot profiles are not specific to minor releases so don't need to change. Don't forget DIY DICE as this has its own installbase and diceinstallbase profiles.