Packages - Sites Mirror

The package repository site mirrors are stored on local disk on the master packages server deneb (an MPU-managed server). There is currently one (virtual disk) partition, /disk/data. Before adding any new site mirrors you MUST check that there is sufficient space on this partition.

The site mirrors do not necessarily contain a complete copy. For instance on EPEL we are not copying any ppc64 directories as we do not need them. On SL and epel we exclude those releases we don't use, leaving only the particular ones we need (e.g. sl72, sl73, epel7, epel6). It should also be noted that, in the directories we do mirror, we do not delete files which disappear from the upstream site. This is in case we ever need a package (or source) which can no longer be found anywhere else.

We currently have these mirrors:

• CentOS virtualisation packages - /disk/data/centosvirt
• Ceph distributed storage platform - /disk/data/ceph
• EPEL (6 and 7) - /disk/data/epel
• IS (packages from other schools) - /disk/data/is
• PostgreSQL - /disk/data/postgresql
• Scientific Linux (6 and 7) - disk/data/sl - we only mirror those minor releases which we currently need and support.
• Software Collections - /disk/data/scl

Adding new mirrors

The mirroring is done by the mirror_sites script. This lives in the scripts subdirectory of the pkglist-tools package (in svn.lcfg.org). The configuration for mirror_sites - the mirror definitions - is in the conf subdirectory of pkglist-tools. The main configuration file is called site_mirrors.ini. The syntax of the configuration file is described in the help message of the mirror_sites script:

[deneb]cc: /usr/share/pkglist-tools/scripts/mirror_sites --help

Some of the current site definitions include other files containing extra snippets of configuration. These other files can also be found in the conf directory of pkglist-tools.

Building and installing the package

Once the changes have been made and committed, the package rebuilt with a new version number and the new version built and submitted to the package servers, the package version will need to be updated in the lcfg/options/pkglist-tools.h header. To get the package installed more quickly you can use live/package-mirror.h whilst waiting for it to filter through the weekly release process. However remember that the primary package server gets its packages from the DR server, so the new version will not be available to updaterpms until the DR server has run its own nightly mirror of the packages. (The DR mirror happens before the primary package server runs updaterpms so the new pkglist-tools should be installed that night.)

Running a mirror sync by hand

The mirror sync process is done every night via cron so if you are happy to wait then everything will happen automatically. If you want to do an initial mirror immediately then this can be done. First, remind yourself of the name of the site to be mirrored:

$ /usr/share/pkglist-tools/scripts/mirror_sites --list

If you have added a new mirror called blob then blob should appear in the output of that command. Then, to sync the blob mirror - always as the pkgmirror user please! - type:

$ nsu pkgmirror
% /usr/share/pkglist-tools/scripts/mirror_sites --site blob