Updating LCFG to a new minor release
The generation of package lists is done using the lcfg-yummy tool, there are some scripts in the
pkglist-tools
package which will do most of the work. For SL6 any x86_64 machine will do, it can be SL6 or SL7 (e.g. your own desktop machine), just add the following to the LCFG profile:
#include <dice/options/pkglist-tools.h>
#include <dice/options/yummy.h>
#include <live/release-scripts.h>
Package Lists
Firstly create a complete set of empty package list files for both the lcfg and dice levels, use a previous minor release as a guide to which files are required. For example, creating sl75 files based on sl74:
% cd core/packages
% for old in {dice,lcfg}/*sl74*.rpms; do \
new=$(echo $old | sed -e 's/sl74/sl75/');\
test -f $new || ( touch $new && svn add $new); \
done
svn commit -m "Added LCFG level package lists for SL7.5" lcfg/*_sl75_*.rpms
svn commit -m "Added DICE level package lists for SL7.5" dice/*_sl75_*.rpms
These files need to exist so that it is possible to test the various package list combinations, without them the various test LCFG profiles will not compile.
Headers
There are a couple of headers which need to be created for each minor release, these can be copied from the previous versions and modified appropriately. For example:
svn cp core/include/lcfg/hidden/sl74.h core/include/lcfg/hidden/sl75.h
There is also a
live/tests-pkglist-sl7.h
header which needs to be updated so that the new package lists can be tested. In the top block ensure that the two new "hidden" files are included. This is what we have for SL7.4:
#include <lcfg/hidden/sl74.h>
#include <lcfg/hidden/sl7vars.h>
Also ensure that the
profile.release
resource in that header has been set to
develop
.
The headers (lcfg, ed, dice and live) must all be searched manually to find all references to the
profile.packages_options
resource. Each one needs to be checked to see if the versions are still correct for the new minor update. Similarly the LCFG profiles for DICE need to be checked. There is no simple way to automate that process so it's all a bit tedious...
Generate the kernel and updates lists
The updates and kernel package lists are generated using the
/usr/share/pkglist-tools/scripts/build_package_lists
script which is driven by the
/etc/pkglist-tools/lcfg_updates.ini
configuration file. There needs to be an entry in this file for each architecture of each minor platform (e.g. sl74 and sl75). The entries for SL7 look like:
[platform sl75]
distro=sl7
releasever=7.5
arch=x86_64
[platform sl74]
distro=sl7
releasever=7.4
arch=x86_64
That file is not managed by LCFG so don't edit the file directly, change the master copy in the pkglist-tools package and create a new package. Once the new package is installed on your machine just run the script in the correct directory in your svn working copy, note that it takes a little while.
% cd svn/lcfg/core/packages/lcfg/
% /usr/share/pkglist-tools/scripts/build_package_lists
The expected output is something like this and should include your new minor platform:
Making sl68 updates package lists
Making sl68_64 updates package lists
Making sl74 updates package lists
Making sl75 updates package lists
Making SL 6.8 kernel package lists
Making SL 7.4 kernel package lists
Making SL 7.5 kernel package lists
Commit the new files into svn, e.g.:
svn commit -m "SL7.5 updates and kernel package lists" lcfg_sl75_updates.rpms lcfg_sl75_kernel.rpms
There may be updates for other platforms, they must be reviewed (check with
svn diff
) before committing. Note that the sorting of packages is annoyingly unstable so sometimes a file changes although there are no actual functional changes.
LCFG Release package
A new minor release needs to be made of the relevant lcfg-release package (e.g.
lcfg-release-el7
):
cd svn/source/trunk/lcfg-release-el7/
lcfg-reltool minor
For SL7.5 this gives the following output:
No version-control information in the LCFG metafile for this project.
Auto-detected that the SVN module should be used.
Checking that all file changes have been committed.
Generating Changelog from subversion log
Updating build ID file
Tagging lcfg-release-el7 at version 7.5.0
Build the package and then submit the SRPM for building by pkgforge in the
lcfg
bucket for the correct platform:
pkgforge submit -B lcfg -p el7 /afs/inf.ed.ac.uk/user/s/squinney/lcfgbuild/lcfg-release-el7-7.5.0/lcfg-release-el7-7.5.0-1.src.rpm
Once built the package should be added to the list in the
lcfg/defaults/profile.h
header:
#ifdef LINUX_EL7
#if OS_RELEASE_MINOR == 3
!profile.packages mEXTRA(lcfg-release-el7-7.3.0-1/noarch:br)
#elif OS_RELEASE_MINOR == 4
!profile.packages mEXTRA(lcfg-release-el7-7.4.1-1/noarch:br)
#else /* 7.5 */
!profile.packages mEXTRA(lcfg-release-el7-7.5.0-1/noarch:br)
#endif
#endif /* LINUX_EL7 */
The
:br
flags are essential as this is how we guarantee that a reboot will occur when a machine updates to the new minor release.
Update yummy configuration
The yummy configuration files are stored in subversion alongside the packages directory:
% svn co https://svn.lcfg.org/svn/lcfg/core
% ls core
include packages validate yummy
Inside the yummy directory there are sub-directories which match with those in the packages directory.
Due to manual edits, over time the package lists tend to drift away from the yummy templates which were originally used to create the files. The first step is to ensure the yummy files are up-to-date, this can be done on any machine for all platforms and architectures.
The
base and
desktop package lists are architecture specific, the default architecture for the packages must be specified. Take the lists for the current minor release (e.g. SL7.4) and produce the release-independent yummy files.
for i in base desktop; do \
/usr/share/pkglist-tools/scripts/pkglist_to_yummy --arch x86_64 packages/lcfg/lcfg_sl74_$i.rpms \
> yummy/lcfg/lcfg_el7_$i.yummy ;\
done
It is important to note for the
base lists that the kernel package will be missing from the yummy file, you will need to manually add this back to the yummy file.
The following lists are used by both architectures:
for i in installbase installroot international kernel kvm options; do \
/usr/share/pkglist-tools/scripts/pkglist_to_yummy packages/lcfg/lcfg_sl74_$i.rpms \
> yummy/lcfg/lcfg_el7_$i.yummy;\
done
Each of the
base
,
installroot
and
installbase
yummy files will need a
kernel
package entry adding to satisfy dependencies. They don't appear in the LCFG package lists since the kernel package list is in a separate package list which is included.
The differences in the yummy files now need to be examined (using
svn diff
), most of them will just be additions of extra packages. If all looks good the changes can be committed.
Package List Creation
The generation of the package lists is done in stages, each stage requires the lists generated by the previous stage to be correct.
Stage 1 : Generate the base package lists
The generation of the base, installbase and installroot package lists are done in stage 1:
/usr/share/pkglist-tools/scripts/make_sl7_stage1 7.5
Replace
7.5
with the required minor release version. If all goes well, the output from the script will look something like:
Loaded plugins: kabi, priorities
Loading support for Red Hat kernel ABI
291 packages excluded due to repository priority protections
warning: Generating 12 missing index(es), please wait...
Generated /disk/scratch/lcfgpkgs_9XCD2w/lcfg_sl75_base.rpms
Loaded plugins: kabi, priorities
Loading support for Red Hat kernel ABI
291 packages excluded due to repository priority protections
Generated /disk/scratch/lcfgpkgs_9XCD2w/lcfg_sl75_installbase.rpms
Loaded plugins: kabi, priorities
Loading support for Red Hat kernel ABI
291 packages excluded due to repository priority protections
Generated /disk/scratch/lcfgpkgs_9XCD2w/lcfg_sl75_installroot.rpms
Results are in /disk/scratch/lcfgpkgs_9XCD2w
If any of the yummy runs failed then the yummy configs will need to be editted in your svn working copy and the changes submitted. Continue to iterate until the package lists are all successfully generated.
If successful, copy the generated package lists into the appropriate location in your svn working copy and commit the changes.
For the installroot the latest
kernel
and
linux-firmware
packages will need to be added to the
lcfg_el7_lcfg_installroot.rpms
package list (you can get the versions from the
lcfg_sl75_kernel.rpms
package list that you generated earlier. The associated
kernel-install-sl7
package can be built later. It should look something like this:
#else /* 7.5 onwards */
kernel-3.10.0-862.3.2.el7
linux-firmware-20180220-62.git6d51311.el7/noarch
#endif
The new package lists must next be tested to ensure they work with updaterpms and do not conflict with other headers and package lists. Firstly check on the
LCFG test server status page that the relevant LCFG profiles have compiled. The profile names have a
sl6-pkglist-
or
sl7-pkglist-
prefix, at this stage the profiles of interest are 1, 2, 3, 4, 9, 10.
If they have all compiled then run the test script for each platform (
sl7
) like this,
as root:
/usr/lib/lcfg/release-scripts/scripts/test_package_lists --platform sl7 --tests 1,2,3,9,10
Check for any errors and modify any package lists (e.g.
lcfg_el7_lcfg.rpms
) and headers where necessary.
Stage 2 : Generate the desktop and options package lists
The generation of the desktop, options and kvm package lists are done in stage 2 (also kvm, scl and latex for SL7):
/usr/share/pkglist-tools/scripts/make_sl6_stage2 6.6
Replace
6.6
with the required minor release version. If all goes well, the output from the script will look something like:
The chroot for profile 'sl6-pkglist-01' is /tmp/lcfgpkgs_tweWp1/lcfgtests/sl6-pkglist-01
Loaded plugins: kabi, priorities
Loading support for Red Hat kernel ABI
380 packages excluded due to repository priority protections
Generated /tmp/lcfgpkgs_tweWp1/lcfg_sl66_desktop.rpms
The chroot for profile 'sl664-pkglist-01' is /tmp/lcfgpkgs_tweWp1/lcfgtests/sl664-pkglist-01
Loaded plugins: kabi, priorities
Loading support for Red Hat kernel ABI
487 packages excluded due to repository priority protections
Generated /tmp/lcfgpkgs_tweWp1/lcfg_sl66_64_desktop.rpms
Loaded plugins: kabi, priorities
Loading support for Red Hat kernel ABI
487 packages excluded due to repository priority protections
Generated /tmp/lcfgpkgs_tweWp1/lcfg_sl66_options.rpms
Loaded plugins: kabi, priorities
Loading support for Red Hat kernel ABI
487 packages excluded due to repository priority protections
Generated /tmp/lcfgpkgs_tweWp1/lcfg_sl66_kvm.rpms
Results are in /tmp/lcfgpkgs_tweWp1
If any of the yummy runs failed then the yummy configs will need to be editted in your svn working copy and the changes submitted. Continue to iterate until the package lists are all successfully generated.
If successful, copy the generated package lists into the appropriate location in your svn working copy and commit the changes.
The new package lists must next be tested to ensure they work with updaterpms and do not conflict with other headers and package lists. Firstly check on the LCFG test server status page that the relevant LCFG profiles have compiled. The profile names have a
sl6-pkglist-
or
sl664-pkglist-
prefix, at this stage the profiles of interest are 5, 6, 7, 8, 14.
If they have all compiled then run the test script for each platform (sl6 and sl664) like this:
/usr/lib/lcfg/release-scripts/scripts/test_package_lists --platform sl664 --tests 5,6,7,8,14
/usr/lib/lcfg/release-scripts/scripts/test_package_lists --platform sl6 --tests 5,6,7,8,14
Check for any errors and modify any package lists (e.g.
lcfg_sl6_lcfg.rpms
) and headers where necessary.
Stage 3 : Generate the international package lists
The generation of the international packages list is done in stage 3:
/usr/share/pkglist-tools/scripts/make_sl6_stage3 6.6
Replace
6.6
with the required minor release version. If all goes well, the output from the script will look something like:
The chroot for profile 'sl664-pkglist-05' is /tmp/lcfgpkgs_48SLZe/lcfgtests/sl664-pkglist-05
Loaded plugins: kabi, priorities
Loading support for Red Hat kernel ABI
649 packages excluded due to repository priority protections
Generated /tmp/lcfgpkgs_48SLZe/lcfg_sl66_international.rpms
Results are in /tmp/lcfgpkgs_48SLZe
If any of the yummy runs failed then the yummy configs will need to be editted in your svn working copy and the changes submitted. Continue to iterate until the package lists are all successfully generated.
If successful, copy the generated package lists into the appropriate location in your svn working copy and commit the changes.
The new package lists must next be tested to ensure they work with updaterpms and do not conflict with other headers and package lists. Firstly check on the LCFG test server status page that the relevant LCFG profiles have compiled. The profile names have a
sl6-pkglist-
or
sl664-pkglist-
prefix, at this stage the profiles of interest are 15.
If they have all compiled then run the test script for each platform (sl6 and sl664) like this:
/usr/lib/lcfg/release-scripts/scripts/test_package_lists --platform sl664 --test 15
/usr/lib/lcfg/release-scripts/scripts/test_package_lists --platform sl6 --test 15
Check for any errors and modify any package lists (e.g.
lcfg_sl6_lcfg.rpms
) and headers where necessary.
Installer kernel
The kernel used in the LCFG installer for the platform should be updated to the most recent available. See
LCFGLinuxInstall for details.
Installbase profiles
New
installbase
and
diceinstallbase
profiles should be generated, and tested with installs. The
installroot
profiles are not specific to minor releases so don't need to change. Don't forget DIY DICE as this has its own
installbase
and
diceinstallbase
profiles.