LCFG Annual Review 2013
On Thursday 5th December 2013 instead of our normal monthly Deployers Meeting we will be holding an Annual Review session.
All users of LCFG are encouraged to attend this meeting to hear about what has been happening over the last year and what developments they can look forwards to in the next year. This is also an excellent opportunity to raise issues that are important to you, put forward ideas for future developments you would like to see and chat about all things LCFG!
If you have any topics you are particularly keen to have discussed then please edit this page and add them to the General Discussion section below with a brief summary.
This will start at 2pm and we aim to be finished by 5pm. It will be held in room 2.33 of the Informatics Forum (note that this is NOT the usual room).
The rough outline agenda is:
14:00 - 14:30 |
Upstream review from Informatics |
14:30 - 15:00 |
Downstream review from IS |
15:00 - 15:30 |
Tea break (mince pies if you're lucky!) |
16:00 - 17:00 |
General Discussion |
After the meeting there will be an informal gathering in a local pub followed by some food, everyone is welcome to come along.
Upstream
LCFG Client Update
The major work for this year was a refactoring of the LCFG client code base. This follows the succesful work done on the server last year. Full details of the
project plan are available
, Stephen also
blogged
about the project.
- Cleaned up code, improved maintainability
- New test suite
- Eradicated hard-wired paths (can use rdxprof as a normal user)
- Fixed a couple of long-standing bugs:
- Handles DNS changes for servers
- Does not block if network port is unavailable
- New generic profile support
Informatics are likely to be switching to the V3 client early in 2014. We may be able to maintain the capability to run with the V2 client for the short-term but it's unlikely to see further support.
The
next phase of work
has now begun. We are working on dropping the dependency on the ancient, and very unloved, W3C XML modules and moving to the standard, well-supported
XML::LibXML
modules. This should be completed within the next couple of months.
Generic Profiles
Once we have updated to the V3 LCFG client we will be able to enable the new support for "generic profiles". This new feature is designed to support two particular scenarios:
- roaming machines
- A roaming machine may regularly change hostname and/or domain name. Previously the LCFG client expected the LCFG profile name to follow the FQDN. We can now support consistently following the same LCFG profile no matter what the hostname is currently. Note that this prevents the receiving of "new profile notifications" but the client still regularly polls anyway.
- many machines with single profile
- Often sites have large sets of identical hosts which do not require any host specific configuration (that comes separately from dhcp, for example). Previously there would have to be a separate profile for each machine which might just be needless duplication, there is now support for the use of a single LCFG profile. Note that this completely breaks the use of spanning maps for managing your network infrastructure but that might not be a problem.
Auto-builder
To help component authors get their software built and submitted into the main lcfg package repository we have added support for auto-building whenever a new version is tagged in the subversion repository (i.e. using one of the
major
,
minor
or
micro
commands for the
lcfg-reltool
app, . This is done using a subversion hook which checks out the version and makes an SRPM which is submitted to the Informatics build-farm. This is primarily aimed at those who do not have DICE accounts and do not have direct access to our PkgForge system. It can be easily enabled by setting the
autobuild
parameter in the
build
section of the
lcfg.yml
file.
Component Changes
Changes to approximately 60 components, 9 of which are new this year:
- lcfg-accessconf
- lcfg-k5login
- lcfg-limits
- lcfg-lsyncd
- lcfg-ocsreports
- lcfg-opennebula
- lcfg-rsnapshot
- lcfg-sensu
- lcfg-update
Smaller stuff:
- updaterpms now reports deletion progress
- ngeneric can configure portreserve and run portrelease
- om - new perl API
LCFG::Om::Command
General Discussion
- Components logging to syslog instead of direct to files by adding support for that to
ngeneric
.
- ngeneric support for other stuff - before methods are called it could set env variables and/or obtain kerberos tickets from keytabs
- Importing resources from databases.
- "Lightweight" use of LCFG, no
boot
component, only managing a few things on the OS.
- Header refactoring
- Switching to git
- SL7
- File component deleting files no longer in the profile.
- XFS, ext4?
- systemd
- Software Collections
SL7
There was a fair amount of discussion regarding systemd. We will need
a new component, if only to configure systemd for the daemons that are
not managed with components. It was suggested that the "boot"
information for components could be moved into the ngeneric namespace,
not clear if this would be of much benefit since we still need to able
to configure systemd anyway. A new systemd component will probably not
take over the whole init process in the way as the current boot
component. It was noted that all components which directly call
scripts in /etc/init.d will need to be updated to use the "service"
command. This could be done now as that works the same way with
upstart.
The question of using XFS as the default partition type was raised. It
seems likely that this will be the RHEL recommended FS. There was
general agreement that we should follow RHEL if possible. Informatics
will have to check if this is compatible with AFS but that wouldn't
stop the default in the LCFG layer being XFS.
There was discussion of the default filesystem layout. Particularly in
the MDP world the third partition is used for /var and the current 5GB
default size is not large enough. It was noted that EL7 could
introduce a new FHS layout with new partitions being required. We will
discuss this further once we see the first beta release.
Component Frameworks
Various improvements to the LCFG component framework (ngeneric) were
proposed. We identified a general class of problems we would like to
solve which would include tasks such as setting environment variables
and initialising kerberos credentials before component methods are
called. This would be best done with a pluggable architecture so that
different groups can develop what they require without having to rely
on Informatics staff to develop (or at least integrate) the features.
It was mentioned that it is currently quite difficult to develop new
functionality because we have always tried to keep the shell and Perl
frameworks equivalent. The shell framework is already causing us quite
a few problems with issues like file-descriptors not being closed
before daemons are started. There was a general agreement that we
would like to deprecate the shell framework and declare a ban on
creating new shell-based components. It was also agreed that we should
aim to remove the shell framework entirely for SL8 (which is probably
3 years away). This would free us up to redevelop the Perl framework
and add new features which handle most of the common requirements.
There was also agreement that we should prefer the Perl Template
Toolkit template style over the older LCFG-style templates. Currently
the default with sxprof and LCFG::Template is to use LCFG-style
templates. The proposal is that we switch this around so that the Perl
TT is the default. The plan is to change this behaviour with the SL7
release. Older style templates would still be supported for SL7 with
an aim to drop support with SL8.
There was some discussion over the idea of having a new python
framework so that components can be written in that language. This
would need some thought on the best design for the API rather than
just blindly copying the Perl framework. We also need to be sure this
doesn't cause us the same sort of problems as supporting Perl and
shell. This isn't going to happen soon but possibly an aim for the SL8
release would be achievable.
Lightweight LCFG
The idea of taking a "lightweight" approach to LCFG was discussed. The
idea being that a system would just have the client and a very small
number of components installed rather than the whole machine being
fully managed. Something similar was done a few years ago so we know
it is possible. It is likely that systemd should make this easier than
currently possible. Probably just needs a bit of effort to structure
the headers appropriately.
This led onto a discussion of whether it would be possible to support
LCFG on Debian/Ubuntu. There is no technical reason for this to not be
possible but it requires someone to have the necessary spare
time. Particularly we would need buildtools support. Also, lots of
components would need configuration file locations to be moved into
resources (that's something we've been working on for years already).
Security
There was discussion of how to make the delivery of XML profiles to
the clients more secure. It was noted that the current client "profile
fetcher" still needs more work. This shouldn't be too much effort now
we have the refactored V3 client. We really want a fetcher class which
supports the use of pluggable authentication handlers. That way
different groups can use their own preferred authentication mechanisms
(e.g. Kerberos or certificate-based) rather than the default user/pass
system.
It would be nice to be able to secure access to the profile on the
client so that only authorised users could run queries. This is harder
and would probably need consolehelper-style PAM support to be added to
qxprof and other tools. That's not likely to happen any time
soon... One suggestion was that we could add support for securely
retrieving resource values from remote services such as wallet, ldap,
postgres. This would mean passwords and other secrets would not have
to be stored directly in the profile.
File Component
A number of groups would like to see a change in behaviour for the
LCFG file component so that it deleted files when they are no longer
managed. Kenny already has a patch to do this so there is not much
work involved other than applying the patch, building the packages and
testing them.
The agreement was that this would be tested in the MDP world first and
once we're happy that it is working as desired it would be pushed into
the LCFG-level. As this is a major change in behaviour we would prefer
to have it become the default only for SL7 onwards with SL6 staying
with the old file component.
This is a general problem which many components need to
solve. Currently many components are copying and pasting code rather
than sharing functionality. We should look at adding support for a
"file registry" system to the LCFG component framework. This would
allow both the removal of files which are no longer managed and an
optional full-cleanup stage on complete removal of the component
package.
To summarise, the main proposals are listed below. Timescales are based around the delivery of the next major platforms. SL7 will arrive in early 2014. We expect SL8 approximately 3 years ahead (i.e. some time in 2017).
LCFG Core
- Shell components are now deprecated, we will not accept any new ones in the core.
- Support for Shell components will be dropped in SL8.
- New support for writing components in Python will be added.
- The LCFG-style templates are deprecated, the default style in SL7 will be Perl Template Toolkit
- Support for LCFG-style templates will be dropped in SL8.
- Support will be added to the file component in SL7 for deleting files when resources are removed.
- Look at adding support for a generic "file registry" system which can be used by all components which need to track the files that have been created.
- Work on improving the end-to-end security of delivering LCFG resource data.
SL7
- Create new systemd component
- Consider XFS for default filesystem type
- Consider partition layout
- Update components to use "service" command
Misc
- Consider support for lightweight LCFG
-- Main.squinney - 2013-10-11