Package Repository Mirroring

The various package repositories (e.g. SL7, SL6, epel) are automatically mirrored each night by a cron job which runs on the packages server - juice. This is done as a separate user (named pkgwrite).

The mirrors can be brought up-to-date at any time via the following process.

Firstly login to juice and become the pkgwrite local user.

% ssh juice
% nsu pkgwrite

You should ensure there are no other rsyncs already running (sometimes the sync process takes ages).

% ps -upkgwrite

Now run the mirror script.

% /usr/share/pkglist-tools/scripts/mirror_sites

This is in the bash history so you can just search back to find it rather than copy/paste, if you like.

This will produce various output, you should check that none of the rsync commands failed. A normal report would like something like this:

Processing site 'epel'
Stashing packages for site 'site epel'
Processing site 'is_uoe'
Stashing packages for site 'site is_uoe'
Processing site 'is_world'
Stashing packages for site 'site is_world'
Processing site 'pgrpm-93'
Stashing packages for site 'site pgrpm-93'
Processing site 'pgrpm-94'
Stashing packages for site 'site pgrpm-94'
Processing site 'sl'
Stashing packages for site 'site sl'

This can take a while to complete, if you are in a hurry it is possible to limit the mirroring to particular sites by using the --site option, e.g.

% /usr/share/pkglist-tools/scripts/mirror_site --site sl

It is possible to get a list of all available sites by using the --list option:

% /usr/share/pkglist-tools/scripts/mirror_sites --list
Mirror sites available:

For complete help on how to use and configure the mirror script see the help docs:

/usr/share/pkglist-tools/scripts/mirror_sites --help

Once that is all done successfully you can logout from juice.

Updating the Package Lists

The updates should always be done after the testing release has been made on a Monday morning. This gives maximum time for finding problems in the develop release. Additional to that, it's a good idea to check for updates again on the Wednesday or Thursday. Applying updates on a Friday is not a good idea since it doesn't give enough working time before the next testing release is made. Of course, if there is a critical security update that must go out immediately then that overrides any worries about not having sufficient time to test. In that case COs should be warned about the issue and extra testing should be done immediately.

To run the scripts you will need to have a checked-out copy of the directory which contains the lcfg package lists. If you don't already have this accessible somewhere then do the following:

% svn checkout
% cd core/packages/lcfg

If you already have a checked-out copy then firstly ensure it is up-to-date and you don't have any outstanding changes to commit (in case they get overwritten).

% svn update
% svn status

To update the package lists you need the pkglist-tools package installed on your machine. Check the dice/options/package-mirror.h header for the latest version (as of September 2012 it is 1.0.63). There is a single script which will do the necessary work:

% /usr/share/pkglist-tools/scripts/build_package_lists

This will modify quite a few package lists, note that you do NOT want to actually commit all of the changes! The important changes are those for the various lcfg_*_updates.rpms package lists.

You should review the changes to the updates package lists. It's best to do each platform separately, this means that if a change on one platform needs to be reverted then it's a much simpler task.

% svn diff lcfg_sl5*updates.rpms | less
% svn commit -m "SL5 updates up to $(date)" lcfg_sl5*updates.rpms
% svn diff lcfg_sl6*updates.rpms | less
% svn commit -m "SL6 updates up to $(date)" lcfg_sl6*updates.rpms

You should check that the updates look consistent across architectures (i386 and x86_64) and, when we're managing multiple minor versions for a platform, the minor releases (e.g. 6.2 and 6.3). Very occasionally an update won't have been completely mirrored for all versions. This might require a bit of hand-editing to make things match (or just wait for the update to be fully available before committing any changes).

They should match up with the security errata messages which SL send out. It's worth noting any big changes or anything you think might cause problems so you can warn COs.

When reviewing the changes check to see if there are any changes to the kernel packages. If not you can just revert modifications to the kernel packages, e.g.

% svn revert lcfg_sl5*kernel.rpms
% svn revert lcfg_sl6*kernel.rpms

Similarly, on SL5 if there are no kernel updates AND no xen package changes you can just revert the xen lists:

% svn revert lcfg_sl5*xen.rpms

If there are kernel package changes you will need to commit the changes to the kernel package lists. On SL6 this is a trivial step:

% svn commit -m "SL6 kernel updates up to $(date)" lcfg_sl6*kernel.rpms

On SL5 it is a much messier task as quite a bit of hand-editing is required to remove from the files all the kernel-module packages for old kernel versions. It can be mostly done with a bit of Perl.

First find the new kernel version (e.g. 2.6.18-308.13.1.el5) then run this (just carefully replace your kernel version into the string between the \Q and the \E).

% perl -pi -e 's/^kernel-module-\w+-(?!\Q2.6.18-308.13.1.el5\E).+$//s' lcfg_sl5*_kernel.rpms lcfg_sl5*xen.rpms

You will then be left with the files containing the kernel package changes you actually want plus a couple of small modifications related to use of cpp (e.g. #ifdef sections) which must be put back by hand to match where they were previously located.

Once you're happy that all those changes are correct you should submit them:

% svn commit -m "SL5 kernel updates up to $(date)" lcfg_sl5*kernel.rpms lcfg_sl5*xen.rpms

At this point the only modified files remaining should be various postship package lists. You probably won't need these changes but hang on to them until the testing is done.

Testing the Updates

Once all the changes are committed you should check the LCFG servers to see that no profiles are broken (or more broken than they were before) due to package conflicts. This doesn't happen very often and is mostly a problem on SL6 when someone has used the profile.packages resource instead of profile.packages_options or has used the wrong base version for a package.

The next stage is to find develop machines for each platform and architecture and run updaterpms to see what happens. Mostly it will just work but occasionally the updates will pull in new dependencies which must be sorted out via the postship mechanism. You might need to use yum to install the update to work out what new package it requires. That new dependency can then be put into the relevant postship package lists. When adding an entry to the postship list you should add a comment with a date and a note about which package required the new dependency. You will need to do this separately for each minor release and architecture for that platform.

On SL5 the postship package lists used to be completely auto-generated but this got out of control and was causing more trouble than it was worth. The build_package_lists script will still put ALL new packages into the list but we only commit changes when necessary so the modifications are more for helpful guidance than anything else. If you don't need any of the changes then just revert the postship lists to their previous version.

Topic revision: r6 - 21 Jul 2015 - 16:02:07 - StephenQuinney
DICE.MPUOsUpdates moved from DICE.MPUFedoraUpdates on 01 Dec 2009 - 12:14 by ChrisCooke - put it back
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies