Tags: view all tags

Using the LCFG rkhunter component

Rootkit Hunter (rkhunter) is an easy-to-use tool which checks computers running Linux for the presence of rootkits and other unwanted tools. For full details you will need to read the rkhunter website.

The LCFG rkhunter component makes it easy to configure the most commonly needed parameters for the tool and supports running the various methods. The rkhunter component should be added to the LCFG profile for a machine in the following way:

#include <lcfg/options/rkhunter.h>

The default values for the resources are typically set to match the standard settings suggested by the rkhunter team. By far the best reference for the purposes of the various rkhunter options is the default configuration file (/etc/rkhunter.conf) shipped with the rkhunter RPM in the epel repository, note that it is only readable by root. It is strongly recommended that you take a copy of this file before starting the LCFG rkhunter component which will overwrite the file. The rkhunter configuration file shipped with the rkhunter 1.4.0 RPM is attached to this page for easy reference.

The configuration provided in the lcfg/options/rkhunter.h header for SL6 was discovered by running rkhunter on a minimal SL6 install and tweaking it until the complaints about legitimate stuff went away. It will need more work for larger installations and specific daemons (e.g. apache, sendmail). As it stands this config works well for something like an SSH login server.

Conflict with prelinking

Many Linux distributions enable the prelink system which alters ELF shared libraries and binaries in an attempt to speed up application startup time. In our experience this causes all sorts of problems when it comes to monitoring filesystems for unauthorised access. In particular, prelinking can cause big issues for rkhunter. It is theoretically possible to configure rkhunter to cope with the prelink system but we found it difficult to get right and we did not feel there was enough of a speed advantage gained by running prelink so it is disabled on hosts which have rkhunter scanning enabled. The prelink tool can be disabled on LCFG systems in the following way:

!profile.packages                     mEXTRA(-prelink-*-*
                                             -lcfg-prelink-*-*)
!file.files                           mREMOVE(prelink)

Configuring rkhunter

This section is not intended to be a complete description of how to configure every possible rkhunter option. Rather the aim is to cover the most commonly needed tweaks to the standard configuration.

In general the rkhunter option names are mapped directly to LCFG resources by just down-casing the name, e.g. APP_WHITELIST becomes rkhunter.app_whitelist.

It is worth noting that there are two modes in which this component can be used. In the first it will manage the contents of the configuration file, in the second it will not manage the configuration, it will just use the file specified in the rkhunter.configfile resource when methods are called. Using a configuration file stored on the local disk clearly introduces some risk of the rkhunter system being subverted. It also gives an attacker a very helpful in-depth insight into how you monitor your systems and thus reveals any likely weakspots. In Informatics we have the configuration file stored in AFS, the root user on a system can read this file (with the help of a Kerberos keytab and an AFS id) but has no ability to modify the file so we have confidence that the daily run is using the correct configuration. The management of the configuration file is controlled via the boolean resource rkhunter.manageconf.

In the scenario where you have disabled the rkhunter.manageconf resource on a particular server then you will have to go to slightly more effort to update your configuration. It is still possible (and recommended) to use the LCFG component to do this task for you. The best way is to include the component and the desired configuration on a private test machine, then set the rkhunter.manageconf resource to true and modify the rkhunter.configfile resource to point to a secure temporary file. Once you have generated the new configuration file and checked it is valid you can then copy into the correct location for use on your important servers (assuming you personally have the ability to update the real configuration file).

1. Whitelisting Scripts

Any executables files in the standard path (e.g. /bin:/sbin:/usr/bin:/usr/sbin) which are scripts (perl, python, shell, etc) rather than compiled executables will be flagged up as a problem. To avoid warnings about legitimate scripts use the rkhunter.scriptwhitelist resource like this:

!rkhunter.scriptwhitelist             mADD(/usr/bin/GET)

2. Hidden Directories and Files

Any hidden directories (i.e. names starting with a .) will be flagged up as a problem. There are a few legitimate directories named in this way which need to be whitelisted. You can whitelist hidden directories using the rkhunter.allowhiddendir resource like this:

!rkhunter.allowhiddendir              mADD(/dev/.mdadm)

Similarly any hidden files found in the filesystem will be flagged up as a problem. There are quite a lot of hidden files, particularly *.hmac files, which need to be whitelisted. You can whiteliste hidden files using the rkhunter.allowhiddenfile resource like this:

!rkhunter.allowhiddenfile             mADD(/lib*/.libcryptsetup.so.*.hmac)

Notice that for files wildcard matching is allowed.

3. Processes accessing deleted files

Sometimes a running process will deliberately hang on to a file which has been deleted, that file will not be reclaimed by the kernel until the process ends. Sometimes it is just a bug which you cannot fix, in either case you will want to whitelist those processes to avoid warnings, this is done using the rkhunter.allowprocdelfile resource like this:

!rkhunter.allowprocdelfile            mADD(/usr/sbin/hald)

You can also specify the paths to the deleted file to restrict the checking by appending the files as a list using a : separator, that is done like this:

!rkhunter.allowprocdelfile            mADD(/usr/sbin/sshd:/var/tmp/host_0)

It is also possible to use wildcard matching with the names of the deleted files.

4. Normal Files in `/dev`

rkhunter will consider any files which are found in the /dev directory to be a problem if they are not device files (e.g. normal text files). To avoid warnings with legitimate files (udev has a few in there, for instance) you can use the rkhunter.allowdevfile resource like this:

!rkhunter.allowdevfile                mADD(/dev/.udev/queue.bin)

It is also possible to use wildcard matching with the names of the allowed device files.

5. Old versions of software

rkhunter checks the versions of some software which it considers to be important (e.g. openssl or the apache httpd). Occasionally this means unnecessary warnings are generated, particularly on long-lived platforms such as RHEL where old versions are kept up-to-date by regular patching without the version string changing. It is possible to whitelist certain software using the rkhunter.app_whitelist resource, like this:

rkhunter.app_whitelist mADD(openssl:1.0.0)

Notice that the correct version has been specified after the : character, you can whitelist the software entirely by not specifying the version string but that risks the application being replaced by an attacker if an intrusion occurs.

6. Listening Processes

Some processes need to listen on all available network interfaces. You will need to whitelist those specific processes so that they are not flagged up as a problem. The rkhunter.allowproclisten resource should be used like this:

!rkhunter.allowproclisten             mADD(/usr/sbin/lldpd)

7. Phalanx2 rootkit

The most commonly used rootkit on compromised Linux machines is known as Phalanx2. The scan for this rootkit can take quite a long time so it is not enabled by default but it is probably a very good idea to run this check. Note that a false result does not guarantee that it is not installed on your machine, it may well be hidden from view by the attacker if they have managed to load a trojan kernel module. Hopefully rkhunter will spot that issue by other means... To enable the check use the rkhunter.phalanx2_dirtest resource like this:

!rkhunter.phalanx2_dirtest            mSET(true)

8. Remote syslog

If you are using remote syslogging you will need to configure rkhunter to confirm that is permitted, this is done using the rkhunter.allow_syslog_remote_logging resource, like this:

!rkhunter.allow_syslog_remote_logging mSET(true)

9. SSH configuration

rkhunter will check your SSH configuration to ensure that it is reasonably secure. It is highly recommended that you never allow root login via SSH and that you only permit protocol version 2. To avoid warnings you need to configure rkhunter so that it knows what the intended values of the sshd PermitRootLogin and Protocol options should be, this is done using the rkhunter.allow_ssh_root_user and rkhunter.allow_ssh_prot_v1 resources, like this:

!rkhunter.allow_ssh_root_user         mSET(no)
!rkhunter.allow_ssh_prot_v1           mSET(no)

The allowed values are yes, no or unset (i.e. they are not specified in the /etc/ssh/sshd_config file).

10. Which tests to run

It is possible to control the tests which are run in a number of ways. You can enable them all and then just disable a few or disable them all and enable a few. By default all tests are enabled and then the following are disabled: suspscan, hidden_procs, deleted_files, packet_cap_apps and apps. These are disabled because they can take a lot of time and/or resources to complete. In particular, our experience shows that the deleted_files and the packet_cap_apps checks can take hours to run on a large installation.

We discovered that the suspscan is not a good one to run if you also use tmpwatch to clear out old files from temporary directories. tmpwatch uses the access time of a file and every time you run the suspscan test on a temporary directory the access time for all the files gets updated. This means you just keep accumulating old, unneeded temporary files...

Running Regular Checks

The simplest way to running a regular check of your system is to call rkhunter from cron, this can be done using the component's run method. A cron job can be added like this:

!cron.objects            mADD(rkhunter)
cron.run_rkhunter   AUTOMINS 5 * * *

This will create a cron entry for the root user which will call the rkhunter component run method at some time (randomly selected) between 05:00 and 05:59 each day.

This method generates a short summary on stdout, so with cron this means an email will sent to wherever mail for the root user is normally sent. If all goes well the contents of the email will look something like:

System checks summary
=====================

File properties checks...
    Files checked: 635
    Suspect files: 0

Rootkit checks...
    Rootkits checked : 312
    Possible rootkits: 0

Applications checks...
    Applications checked: 5
    Suspect applications: 0

The system checks took: 2 minutes and 5 seconds

All results have been written to the log file (/var/log/rkhunter/rkhunter.log)

No warnings were found while checking the system.

[OK] rkhunter: run

If alternatively you see something like this in the summary:

Warning: Package manager verification has failed:
         File: /etc/rc.d/rc.local
         The file hash value has changed
Warning: The file properties have changed:
         File: /etc/rc.d/rc.local
         Current inode: 169384    Stored inode: 168665
         The file size has changed
         The file modification time has changed
Warning: The command '/etc/rc.d/rc.local' has been replaced by a script: /etc/rc.d/rc.local: POSIX shell script text executable

System checks summary
=====================

File properties checks...
    Files checked: 635
    Suspect files: 1

Rootkit checks...
    Rootkits checked : 312
    Possible rootkits: 0

Applications checks...
    Applications checked: 5
    Suspect applications: 0

The system checks took: 2 minutes and 6 seconds

All results have been written to the log file (/var/log/rkhunter/rkhunter.log)

One or more warnings have been found while checking the system.
Please check the log file (/var/log/rkhunter/rkhunter.log)

[FAIL] rkhunter: Failed to run rkhunter (see logfile)

then you are going to have to go and check whether the changes are legitimate. The best place to start is with the updaterpms log (/var/lcfg/log/updaterpms) to see whether the package(s) owning the changed files have been altered since the last rkhunter scan. You can find out which package owns a file using rpm -qf /path/to/file

If you are happy that all file changes are correct and as expected then you will need to manually run the propupd method to update the file properties database, see below for details.

It is possible to alter various options when the run method is called, see below for more details.

Component Methods

1. run

This method is used to scan the system for rootkits and other potential security issues. By default the full report of the scan will be appended to the file set in the rkhunter.logfile resource (normally /var/log/rkhunter/rkhunter.log) and a short summary will be printed to stdout. The run method supports the following command line options:

logfile: Overrides the rkhunter.logfile resource setting (the file will be appended to if already has contents)
stashlog: Will store the log as a new file (overwrites any previous contents)
configfile: Overrides the rkhunter.configfile resource setting

Any other options will be passed directly through to rkhunter which makes it possible to use things like --disable and --enable to temporarily disable and enable tests. See the manual page for rkhunter(8) for full details.

Note that the run method already sets the following options when running the --check method: --nocolors, --skip-keypress, --report-warnings-only and --summary. Setting other conflicting options might cause you trouble...

You can enable the rkhunter --verbose-logging option by setting the LCFG rkhunter component verbose (-v) option.

In Informatics we call the run method like this:

!cron.additions       mADD(rkhunter)
!cron.add_rkhunter    mSETQ('AUTOMINS 5 * * * /usr/bin/k5start -q -k <%rkhunter.tmpdir%>/krb5_ccache \
                                                 -f <%kerberos.file_krb5keytab%> -u host/<%profile.node%>.<%profile.domain%> -t -- \
              om rkhunter run -- --stashlog /afs/inf.ed.ac.uk/group/mp-unit/security/logs/<%profile.node%>/rkhunter.$(date +\%F-\%T).log')

As well as having the read-only configuration file stored in AFS this stashes the logfile into AFS in a timestamped file. The ACLs on the directory are set such that root has permission only to insert new files, it cannot read the contents of files, modify or delete existing files. The k5start stuff is all about acquiring the necessary Kerberos principal and AFS token.

2. propupd

One of the features of rkhunter is that it stores the details of various attributes for files (e.g. timestamp, permissions, hash of contents). Every time a check (via the run method) is carried out this list of files is checked to see if anything has changed. Any changes are reported as warnings in the summary. If you are sure that all the changes are legitimate (e.g. due to a run of updaterpms) then you can update the stored data using this method. Normally all you need to do is this:

om rkhunter propupd

Depending on the size of your installation, this will take a few minutes. If it worked correctly you will get output like:

[INFO] rkhunter: [ Rootkit Hunter version 1.4.0 ]
[INFO] rkhunter: File updated: searched for 167 files, found 635
[INFO] rkhunter: Successfully updated stored properties list
[OK] rkhunter: propupd

If you are interested in exactly what has been done you can find the details appended to the logfile (as specified in the rkhunter.logfile resource).

You can override the configuration file (as specified in the rkhunter.configfile resource) via the --configfile command line option. Any other options set on the command line will be passed directly through to rkhunter.

You can enable the rkhunter --verbose-logging option by setting the LCFG rkhunter component verbose (-v) option.

3. update

Calling this method will cause rkhunter to check if there is a later version of any of its text data files. A command-line web browser, for example wget or lynx, must be present on the system when using this option. In the manpage for rkhunter it is suggested that this command option is run regularly in order to ensure that the data files are kept up to date. In our experience these files rarely change, maybe running update once a month might be sufficient.

Normally all you need to do is this:

om rkhunter update

If you are interested in exactly what has been done you can find the details appended to the logfile (as specified in the rkhunter.logfile resource).

You can enable the rkhunter --verbose-logging option by setting the LCFG rkhunter component verbose (-v) option.

-- Main.squinney - 2013-02-28