Tags:
create new tag
view all tags

Optional Packages from the Main Distribution Repository

There are now two new ways of adding optional software packages in LCFG. Both are clearer and simpler to use than previous methods, and should produce less conflicts, less mess and smaller sets of packages. This page explains both of the new methods.

In the past (up to and including SL5) our strategy for generating the base package lists was to take everything which was in the main package repository for the distribution. In SL5 this translates into a set of 2600 packages which are installed on every DICE machine. There is no doubt that this approach to generating the base package lists requires minimal effort but it leads to a number of issues. This approach makes non-desktop installs much larger than necessary which leads to long install times, wasted disk space and the potential for increased risk of security holes. This also means that there are effectively no optional packages to be added from the base SL5 distribution repository.

From F13 onwards we introduced a separation of "base" and "desktop" package lists, we also stopped installing every package available in the distribution onto DICE machines. This was because the F13 main package repository is immense (nearly 17000 packages) and it's probably not possible to install them all even if the space is available. The F13 base package list contains roughly 400 packages and the desktop list contains 1600 packages which are installed on top of the base. This approach has worked well on the few servers we have installed and results in much faster install times. There are some rough edges though, mostly related to how packages are managed in the DICE layer which result in a lot of conflicts whenever updates are applied. There is also the problem of adding extra packages from the base distribution repository (which are not already in the base list) and then ensuring that they continue to receive updates, preferably in an automated fashion.

To get around these problems of large package lists IS have, for a long time, preferred to instead use the installbase lists. These lists are designed as a stepping-stone in the install process where just enough packages are installed onto a machine for LCFG to take over and complete the installation through the normal package and configuration management processes. The current LCFG SL5 installbase package lists contain only 270 packages. To supplement this small set of packages IS have used package lists containing sets of "options". Full details of the IS optional packages system are on their wiki.

These options are activated by setting extra CPP macros, via the LCFG headers, prior to the pre-processing stage for the package lists. The package list of options would look something like this snippet taken from the IS mdp_sl55_min.rpms package list.

#ifdef NFSCLIENT
nfs-utils-1.0.9-44.el5
nfs-utils-lib-1.0.8-7.6.el5
libgssapi-0.10-2
#endif /* NFSCLIENT */

#ifdef OPENLDAPSERVER
openldap-servers-2.3.43-12.el5
libtool-ltdl-1.5.22-7.el5_4
#endif /* OPENLDAPSERVER */

#ifdef SAMBASERVER
samba-3.0.33-3.28.el5
perl-Convert-ASN1-0.20-1.1/noarch
cups-libs-1.3.7-18.el5
libtiff-3.8.2-7.el5_3.4
gnutls-1.4.1-3.el5_4.8
#endif /* SAMBASERVER */

To enable a particular package list option an extra CPP macro is set via the LCFG headers, like this:

!profile.pkgcppopts     mADD(-DSAMBASERVER)

Note the use of the -D prefix which is the cpp command-line option for defining a macro.

A single host may enable multiple options which include packages with the same name. This works fine as long as the packages are all of the same version, the LCFG server ignores multiple specifications of a package where they are identical. A conflict will arise (and the profile will fail to compile) however if they are for different versions. This should never be a problem as the packages in the options header should only ever refer to packages which are in the distribution base repository (or in the case of new packages shipped later and are only in the updates repository, use the earliest version in the updates repository).

This approach taken by IS is very effective and results in much smaller non-desktop installs. For SL6 we are planning to add a new options package list at the LCFG level, this will be named lcfg_sl60_options.rpms. We do not intend to add a separate x86_64 package list, hopefully we can survive with just a single list. It will be updated at each minor release, hopefully it will be fairly easy to script the generation of the list. We have also agreed on a standard macro namespace for these options which will be LCFG_OPTIONS_, e.g. for apache this would be LCFG_OPTIONS_APACHE.

There is one further complication to this method. It is also necessary to have the standard set of distribution updates applied automatically to the optional sets of packages. This can be achieved by ensuring the options package lists appear before the updates packages lists in the profile.packages resource. This problem has been tackled in a number of ways within the LCFG headers but none were totally satisfactory, for example:

!profile.packages       mSUBST(@lcfg_f13_postship.rpms,@lcfg_f13_devel.rpms @lcfg_f13_postship.rpms)

This places the F13 devel list before the postship list (and thus also the updates list). This has the disadvantage of requiring knowledge of the platform name and it gets even worse for SL5 and SL6 where the minor version is also encoded into the package list file name and the architecture also changes the file name for x86_64. Due to recent changes made to the LCFG sysinfo component it is possible to do this in a general fashion:

!profile.packages       mSUBST(@lcfg_<%%sysinfo.os_id_full%%>_postship.rpms,\
                               @lcfg_<%%sysinfo.os_base%%>_devel.rpms @lcfg_<%%sysinfo.os_id_full%%>_postship.rpms)

This does mean that it is not necessary to add new mutations for every new platform, minor version, architecture, etc., but it has the alternative disadvantage of being utterly incomprehensible to the casual reader. Whether with the specific or general methods shown here it is very unclear why one package list is being replaced with another and itself. There is little to guide the reader as to why this mutation is useful. It is clear from these examples that a much better method is needed.

As well as supporting the addition of packages and package lists to the profile.packages resource the LCFG server allows the use of LCFG "tag" resources. This means that it is possible to do the following:

!profile.packages          mADD(foobar)
!profile.packages_foobar  mEXTRA(foo-1-2/noarch
                                 bar-3-/noarch
                                 @mylist.rpms)

The contents of the profile.packages_foobar resource are inserted into the profile.packages resource at the location of the tag when the list of packages is processed. IS have also made use of this feature for quite a while by inserting an mdpextra tag resource into the list before the entry for the updates package list. This allows users to add packages from the distribution in the LCFG headers and have updates applied automatically.

At the LCFG layer, for all platforms, we have added a new tag resource into the profile.packages named options. This appears before the postship (and thus updates) package lists. As described above, this provides a facility for adding individual packages via the headers but it can also be used for adding the various extra, optional, package lists and getting updates applied. The previous example of the devel package list now becomes:

!profile.packages_options     mEXTRA(@lcfg_<%%sysinfo.os_base%%>_devel.rpms)

There is no doubt that this is much tidier and clearer than the previous example.

Package version selection

When adding extra packages to the profile.packages_options resource you must select the lowest version supplied for that specific minor release of the distribution. So, if the current distribution is SL6.1 select the original version of the package supplied for that minor release, ignore any later updates. The LCFG server ignores additional specifications for a package if the versions match and thus there will be no conflict. Once a package set has been added to the profile.packages_options resource in a core LCFG or DICE header the LCFG fairies will ensure it keeps working for all subsequent minor release updates.

Note that you must not use version wildcards for packages in the profile.packages_options resource. This could easily lead to unexpected and difficult to resolve conflicts at some later date.

Summary

To summarise all this, with SL6 you can now add packages which are supplied as part of the SL6 distribution either to the profile.packages_options resource like this:

!profile.packages_options            mEXTRA(httpd-2.2.15-5.sl6
                                            httpd-manual-2.2.15-5.sl6/noarch
                                            httpd-tools-2.2.15-5.sl6
                                            mod_ssl-2.2.15-5.sl6)

or the package groups can be added to the lcfg_sl60_options.rpms package list like this:

#ifdef LCFG_OPTIONS_APACHE
httpd-2.2.15-5.sl6
httpd-manual-2.2.15-5.sl6/noarch
httpd-tools-2.2.15-5.sl6
mod_ssl-2.2.15-5.sl6
#endif /* LCFG_OPTIONS_APACHE */

and can be enabled in an LCFG header like this:

!profile.pkgcppopts     mADD(-DLCFG_OPTIONS_APACHE)

Updates will be applied automatically in either case.

It is expected that in most cases the package lists will be generated and tested in a header using the profile.packages_options resource and once they are known to be correct they will be transferred to the options package list. Informatics and IS intend to collaborate on creating/managing the sets of packages in the options list to avoid duplication of effort.

Extra Details

For those interested here are some extra details.

The basic setting of profile.packages (in lcfg/defaults/profile.h) for Linux platforms used to look like:

profile.packages      @lcfg_/**/LCFG_PLATFORM_FULL/**/_base.rpms\
                      @lcfg_/**/LCFG_PLATFORM_FULL/**/_postship.rpms\
                      @lcfg_/**/LCFG_PLATFORM_FULL/**/_updates.rpms\
                      @lcfg_/**/LCFG_PLATFORM_FULL/**/_override.rpms\
                      @lcfg_/**/LCFG_PLATFORM_FULL/**/_kernel.rpms\
                      @lcfg_/**/OS_BASIS/**/_lcfg.rpms

With the addition of the new resource and the new header it becomes:

profile.packages       @lcfg_/**/LCFG_PLATFORM_FULL/**/_base.rpms\
                       options\
                       @lcfg_/**/OS_ID_FULL/**/_options.rpms\
                       @lcfg_/**/LCFG_PLATFORM_FULL/**/_postship.rpms\
                       @lcfg_/**/LCFG_PLATFORM_FULL/**/_updates.rpms\
                       @lcfg_/**/LCFG_PLATFORM_FULL/**/_override.rpms\
                       @lcfg_/**/LCFG_PLATFORM_FULL/**/_kernel.rpms\
                       @lcfg_/**/OS_BASIS/**/_lcfg.rpms

#ifndef LCFG_PROFILE_PACKAGES_NO_OPTS
profile.packages_options          /* INTENTIONALLY EMPTY */
#endif

Note that due to some restrictions on the way the profile.packages_options can be used it must be defined (in the Perl sense) but it does not need to contain anything more than the empty string so we have to initialise the resource in case nothing is added.

Note also that the options header is referred to using the OS_ID_FULL macro instead of the LCFG_PLATFORM_FULL, the difference is that the platform macro also refers to the processor architecture in the case of x86_64 (e.g. f13 compared to f13_64 or sl55 compared to sl55_64).

These changes have some big implications for LCFG profiles which create their own package sets from scratch. This mainly affects the LCFG installroot, installbase and initramfs families of profiles. If the options resource is not going to be used at all then the LCFG_PROFILE_PACKAGES_NO_OPTS macro can be set prior to the inclusion of the lcfg/defaults/profile.h header. If the options resource is disabled then care must be taken to not include any LCFG headers which expect it to exist, use of the resource in this situation could easily lead to a profile compilation error. This is probably only acceptable in a very few cases such as that for the installer initramfs which really does only need 5 very specific packages. Similarly, if the options package list is not included then care must be taken to not include subsequent headers which expect it to be included as the setting of options will have no effect (but at least will not cause a profile compilation error).

-- Main.squinney - 2011-05-10

Topic revision: r4 - 2012-04-19 - squinney
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback