Tags:
create new tag
view all tags

Getting Started with OpenAFS & LCFG

Before getting started, if you already have the LCFG openafs component installed on the machines you will be using for the AFS servers then you must stop the component now. The first stages require manually running some of the AFS services without authentication. If the component is running it might end up fighting you all the way.

Note that until otherwise stated all commands should be issued as the root user.

If you get stuck or just want to understand what is going on you can get more information from the various utilities. For example:

% pts help createuser
% bos help stop

You can also get more extensive information from the manual pages, just concatenate the utility name with the command e.g.

% man pts_createuser
% man bos_stop

There are also man pages on most of the files involved (e.g. KeyFile and CellServDB) and all the afs daemons/servers (e.g. afsd, bosserver, vlserver).

All the documentation is also available online at the OpenAFS website.

All the instructions on this page are based on using the RPMs shipped by the OpenAFS project themselves. Other distributions may have the files in different locations and you may need to tweak more than just the LCFG resources mentioned below. See the lcfg-openafs man page for a complete coverage of what can be configured.

Throughout this documentation the Kerberos realm is "EXAMPLE.ORG", the domain is "example.org" and the AFS cell name is "afstest.example.org". Note that normally you would want your AFS cell name to be the same as your Kerberos realm (unless you are setting up a test cell).

Kerberos

If you do not have a Kerberos infrastructure then you should go away now and get that working first. The first thing you need to do is add a Kerberos principal to the KDC and then extract it into a keytab. If you do not have admin rights you will need to go and speak nicely to the relevant person to do this step for you.

$ /usr/bin/kadmin -q "ank -randkey afs/afstest.example.org"
$ /usr/bin/kadmin -q 'ktadd -e "aes256-cts-hmac-sha1-96:normal aes128-cts-hmac-sha1-96:normal des3-hmac-sha1:normal arcfour-hmac-md5:normal"\
                                           -k /usr/afs/etc/rxkad.keytab  afs/circlevm0.inf.ed.ac.uk'

If you were successful you will see some output like this:

Entry for principal afs/circlevm0.inf.ed.ac.uk@INF.ED.AC.UK with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:rxkad.keytab.
Entry for principal afs/circlevm0.inf.ed.ac.uk@INF.ED.AC.UK with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:rxkad.keytab.
Entry for principal afs/circlevm0.inf.ed.ac.uk@INF.ED.AC.UK with kvno 3, encryption type des3-cbc-sha1 added to keytab WRFILE:rxkad.keytab.
Entry for principal afs/circlevm0.inf.ed.ac.uk@INF.ED.AC.UK with kvno 3, encryption type arcfour-hmac added to keytab WRFILE:rxkad.keytab.

Note that you should ONLY do the ktadd once. on one machine, you must NOT do the ktadd on every machine as the key will be regenerated each time and the kvno will change. The keytab must be securely copied to any further servers. The keytab file should be mode 0600 and owner/group root:root (those are the defaults when using ktadd).

AFS DB Servers

The next step is to get an AFS DB server running.

Basic Cell Information

Firstly, set the cellname:

$ bos setcellname -server host1.example.org -name afstest.example.org -noauth

If your cell name does not match your kerberos domain (case-insensitive) then you will want to do this next step as well:

$ echo EXAMPLE.ORG > /usr/afs/etc/krb.conf

Whilst you are at it you can do the same for the client:

$ echo EXAMPLE.ORG > /usr/vice/etc/krb.conf

You should now take the opportunity to educate the client-side about your new AFS cell. You can do this by using the server CellServDB and ThisCell files (which are helpfully generated).

$ cat /usr/afs/etc/CellServDB >> /usr/vice/etc/CellServDB
$ cp /usr/afs/etc/ThisCell /usr/vice/etc/ThisCell

The openafs component will take care of this in a better way later but this is sufficient for now.

The admin users

To manage users and groups you need to start the AFS Protection Server (ptserver). This is done via the bos command:

$ bos create -server host1.example.org -instance ptserver -type simple -cmd /usr/afs/bin/ptserver -cell afstest.example.org -noauth

You can now create the admin users:

$ bos adduser -server host1.example.org -user fred.admin -cell afstest.example.org -noauth
$ pts createuser -name fred.admin -cell afstest.example.org -noauth
$ pts adduser -user fred.admin -group system:administrators -cell afstest.example.org -noauth

This adds your admin user to the AFS super users list, adds it to the protection service and then includes it in the system:administrators group. You can verify that it all worked:

$ pts membership fred.admin -cell afstest.example.org -noauth
$ pts membership system:administrators -cell afstest.example.org -noauth

The next stage will involve using the component but first we need to stop the running processes. Use bos to stop the protection server:

$ bos stop -server host1.example.org -instance ptserver -cell afstest.example.org -noauth
$ bos delete -server host1.example.org -instance ptserver -cell afstest.example.org -noauth

To stop the bosserver you need to find the PID and then use kill. It's important that all the afs processes are stopped or the LCFG openafs component will get quite upset so it is worth checking. This should return nothing:

$ ps aux | grep afs | grep -v grep

Using the openafs Component

1. Add the openafs client and server headers to your LCFG profile:

#include <lcfg/options/openafs-client.h>
#include <lcfg/options/openafs-server.h>

Once you've finished configuring everything for your cell it might be a good idea to remove the AFS client from all the AFS servers. This would avoid any potential for problems when restarting the services.

2. Specify the details of your local cell. This will set up the client configuration but the list of cell DB server information will also be re-used for the server configuration. Note the setting of the krb_realms resource as the cell name "afstest.example.org" does not equal the Kerberos realm "example.org". If your cell information is already in the client CellServDB file as distributed by the OpenAFS project you do not need to add the tag to the openafs.cellservdb_local resource. You do still need to define all the cell information so that the server version of CellServDB can be configured correctly. Note that you must give the name and the IP address for each server.

!openafs.cellservdb_local              mADD(afstest)
openafs.cellname_afstest               afstest.example.org
openafs.cellcomment_afstest            My Test Cell
openafs.cellservers_afstest            afsdb0

openafs.cellserver_addr_afstest_afsdb0 129.215.24.207
openafs.cellserver_name_afstest_afsdb0 host1.example.org

openafs.thiscell                       <%openafs.cellname_afstest%>

!openafs.krb_realms                    mADD(EXAMPLE.ORG)

3. Set the server cell name (that is, the LCFG tag you used above)

openafs.cellservdb_server              afstest

4. Set options for the bosserver, you almost certainly want to allow dotted-principals. This maps the Kerberos foo/admin to foo.admin for AFS.

!openafs.allow_dotted_principal        mSET(true)

5. Add admin users. Note that this does not actually create the users in the protection database or add them to the system:administrators group, you have to do that bit by hand.

openafs.admin_users                    fred.admin

6. Set up the services you want to run, in this case just the protection server and the volume location server. You might also need things like buserver, they can be configured in a similar way.

!openafs.services                        mADD(vlserver)
openafs.service_start_vlserver           yes

openafs.service_commands_vlserver        vlserver
openafs.command_args_vlserver_vlserver   -allow-dotted-principals

!openafs.services                        mADD(ptserver)
openafs.service_start_ptserver           yes

openafs.service_commands_ptserver        ptserver
openafs.command_args_ptserver_ptserver   -allow-dotted-principals

After submitting all those resource changes check that the LCFG profile has compiled correctly. Once the changes have been received by the LCFG client on your AFS DB server you should do:

$ om updaterpms run
$ om openafs configure

You will get a warning about being unable to configure the admin users list but that is ok as the component has not yet been started. If you feel inclined you can check the generated files for the client in /usr/vice/etc and for the server in /usr/afs/etc. If they all look good you can now start the component:

$ om openafs start

You should now have a running AFS DB server ready for service!

To check that all is well, drop out of the root account, kinit as your admin principal and get afs tokens with aklog:

% kinit fred/admin
% aklog
% pts membership system:administrators 
Members of system:administrators (id: -204) are:
  fred.admin

You should now also be able to create more users:

% pts createuser -name fred -cell afstest.example.org
User fred has id 2

Multiple DB Servers

It is considered good practice to have multiple AFS DB servers for redundancy. Having setup the first DB server it is now trivial to add further servers by just adding the LCFG resources from above to more profiles and starting the openafs component. You need to specify the information for all the DB servers in the cell, for example:

openafs.cellname_inf               example.org
openafs.cellcomment_inf            School of Informatics, University of Edinburgh
openafs.cellservers_inf            afsdb0 afsdb1 afsdb2

openafs.cellserver_addr_inf_afsdb0 129.215.64.16
openafs.cellserver_name_inf_afsdb0 afsdb0.example.org

openafs.cellserver_addr_inf_afsdb1 129.215.64.17
openafs.cellserver_name_inf_afsdb1 afsdb1.example.org

openafs.cellserver_addr_inf_afsdb2 129.215.64.18
openafs.cellserver_name_inf_afsdb2 afsdb2.example.org

Once you have them running you can check that they are all talking to each other by using udebug for each service on each host. For example:

% udebug afsdb0 ptserver
% udebug afsdb1 vlserver

You should see that one host (usually the one with the lowest IP address) will have been voted the master in each case.

Monitoring

The LCFG openafs component comes with a nagios translator which can be used to generate nagios configurations from the LCFG XML profile for each host. These use the AFS monitoring scripts supplied by Russ Allbery.

Other Stuff

These bits are beyond the configuration capabilities of the standard LCFG components.

  1. You probably want to add firewall holes for some of the udp and tcp ports 7000 through to 7009, check /etc/services to find out which ones you need.
  2. You also probably want to add some DNS records like these below, this avoids every client needing to have the cell info in the CellServDB file (your client will need the afsdb option turned on).

example.org @ 3600 IN AFSDB 1 afsdb0.example.org.
example.org @ 3600 IN AFSDB 1 afsdb1.example.org.
example.org @ 3600 IN AFSDB 1 afsdb2.example.org.

AFS File Servers

It is possible to add the AFS file server to the same host as the DB server but usually it is considered good practice to use different machines. You can have as many fileservers as you like which allows for load balancing and read-only copies of volumes can be distributed across them for redundancy.

Configuring the Partitions

The AFS file server looks for partitions mounted on locations which match /vicep* If you have a spare primary partition you could use the LCFG fstab component something like this:

!fstab.partitions_sda      mADD(sda4)

fstab.mpt_sda4             /vicepa
fstab.preserve_sda4    yes
fstab.size_sda4            free
fstab.type_sda4           ext3

If your data is on a SAN or some other place it can be done like this:

!fstab.entries                 mADD(vicepa)

fstab.spec_vicepa        /dev/mpath/3600c0ff000d59aa194b16d4901000000p1
fstab.file_vicepa           /vicepa
fstab.vfstype_vicepa    ext3
fstab.passno_vicepa     2

Due to the way the LCFG fstab component works, if your machine is already installed you will have to create and format the partition by hand if it does not already exist. You will also need to add the necessary entry to /etc/fstab

Configuring the AFS file server

The first section of the LCFG configuration for an AFS file server is the same as for the DB server (you probably want to put this into a shared header file):

#include <lcfg/options/openafs-client.h>
#include <lcfg/options/openafs-server.h>

!openafs.cellservdb_local              mADD(afstest)
openafs.cellname_afstest               afstest.example.org
openafs.cellcomment_afstest            My Test Cell
openafs.cellservers_afstest            afsdb0

openafs.cellserver_addr_afstest_afsdb0 129.215.24.207
openafs.cellserver_name_afstest_afsdb0 host1.example.org

openafs.thiscell                       <%openafs.cellname_afstest%>

!openafs.krb_realms                    mADD(EXAMPLE.ORG)

openafs.cellservdb_server              afstest

!openafs.allow_dotted_principal        mSET(true)

openafs.admin_users                    fred.admin

The fileserver is configured slightly differently in that it is not a "simple" type of service, it consists of 3 separate commands which are run. As before you almost certainly want the allow-dotted-principals option, the others are tweakable and are based on the values used in the School of Informatics, you might need something different.

!openafs.services                        mADD(fs)

/* The order of these commands is important */

openafs.service_commands_fs              fileserver volserver salvager
openafs.service_type_fs                  fs
openafs.service_start_fs                 yes

/*
 *
 * Number of threads to run      (-p)      = 128
 * Number of rx extra packets    (-rxpck)  = 400
 * Redirect clients when queue   (-busyat) = 600
 * Number of cached small vnodes (-s)      = 1200
 * Number of cached large vnodes (-l)      = 1200
 * Number of callbacks           (-cb)     = 100000
 * Number of cached dir blocks   (-b)      = 240
 * Maximum volume cache size     (-vc)     = 1200
 *
 */

/* File Server process */

openafs.command_args_fs_fileserver       -L -p 128 -rxpck 400 -busyat 600 -s 1200 -l 1200 -cb 100000 -b 240 -vc 1200 -allow-dotted-principals

/* Volume Server process */

openafs.command_args_fs_volserver        -p 16 -allow-dotted-principals

/* Salvager process */

openafs.command_args_fs_salvager         -parallel all5

Again, after submitting all those resource changes check that the LCFG profile has compiled correctly. Once the changes have been received by the LCFG client on your AFS file server you should do:

$ om updaterpms run
$ om openafs configure

Setting up Volumes

You now need to become the admin user again and acquire AFS tokens:

% kinit fred/admin
% aklog -c afstest.example.org

Firsly it is necessary to create a couple of standard AFS volumes named root.afs and root.cell. Firstly create the root AFS volume:

% /usr/sbin/vos create -server host2.example.org -partition /vicepa -name root.afs -cell afstest.example.org

If you are not using dynamic-root (see the dynroot option for the AFS client) you should set the permissions on the /afs directory like this:

% fs setacl -dir /afs -acl system:anyuser rl

Next create the root volume for the cell, firstly mount it read-only on a directory named after your cell, set the "read" and "lookup" access controls for everyone, finally mount a separate writable copy on a dotted version of your cell name.

% /usr/sbin/vos create -server host2.example.org -partition /vicepa -name root.cell -cell afstest.example.org
% fs mkmount -dir /afs/afstest.example.org -vol root.cell -cell afstest.example.org
% fs setacl  -dir /afs/afstest.example.org -acl system:anyuser rl
% fs mkmount -dir /afs/.afstest.example.org -vol root.cell -cell afstest.example.org -rw

If all has gone well you now have a working AFS fileserver in which you can now create any volumes you like. Managing those partitions and volumes is beyond the scope of the LCFG openafs component, you should refer to the OpenAFS documentation for further details on where to go from here.

-- Main.squinney - 2009-12-01

Topic revision: r7 - 2014-11-21 - squinney
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback