RJ Systems
Linux System Administration
Home Tech Linux Links Consulting







Valid XHTML 1.0!

Valid CSS!

IPv6 test

OpenLDAP consumer with MIT Kerberos V on Debian squeeze

Introduction

This page describes how to set up an OpenLDAP consumer server (syncrepl) with MIT Kerberos V support for authentication and encryption. It depends on a previously installed OpenLDAP provider server, ldapks1.example.com, which also includes Kerberos support, as well as two Kerberos servers, kdc1.example.com and kdc2.example.com − a master and a slave. At least the master KDC will be necessary for this exercise, but it may be more interesting to use the slave.

It is assumed that the reader is familiar with the concepts involved in the installation and configuration of an OpenLDAP consumer server. However, as opposed to that configuration, this OpenLDAP system will only store Unix user account information for authorization; the task of authentication will be performed by Kerberos, with all username and password combinations being stored in a Kerberos database.

In this example, OpenLDAP is installed on a host running Debian 6.0 (squeeze). If followed properly, the step-by-step process should produce a consumer server with a copy of the DIT from the provider. But before the actual OpenLDAP installation process can begin, it will first be necessary to install the operating system on a new host called ldapks2.example.com. A DNS server must also be available on the network with a zone file to which forward and reverse mappings can be added for this host. After the initial installation of the operating system, make sure these packages are installed on the system as well:

~# apt-get install ntp ntpdate nmap

Afterwards, edit /etc/ntp.conf so that the machine synchronizes to a common NTP server (preferably a local one) and edit /etc/default/ntpdate to use the same host also. Now the installation of the new server can begin.


1. Patching the source

Because of a bug in the current version of slapd, 2.4.23-7, the source code must first be patched. Without it, slapd will return errors regarding the olcDbURI attribute (necessary for the chaining configuration) and even refuse to (re)start if it is configured that way. So, prepare the new host for a package build by installing the following packages on the new host, ldaps2:

~# apt-get install dpkg-dev devscripts

This will result in 100 new packages being installed. Next, obtain the source:

~# cd /usr/src
~# apt-get source openldap

Three files will be downloaded and a directory, ./openldap-2.4.23/, will be created that contains the source code. Soon, seven Debian packages will be built from it, but first the various dependencies for these package must be downloaded and installed:

~# apt-get build-dep openldap

70 new packages will be installed as a result. Next, download and apply the patch:

~# wget -q ftp://ftp.openldap.org/incoming/pierangelo-masarati-2010-04-29-chain.1.patch
~# patch openldap-2.4.23/servers/slapd/back-ldap/chain.c < pierangelo-masarati-2010-04-29-chain.1.patch
patching file openldap-2.4.23/servers/slapd/back-ldap/chain.c
Hunk #1 succeeded at 1137 (offset -3 lines).
Hunk #2 succeeded at 1148 (offset -3 lines).
~# _

Then, enter the source code directory, add a suffix to the Debian version number to distinguish the results of this build, and compile the binary package without signing the .changes file:

~# cd openldap-2.4.23/
~# dch -l patched 'Including a patch by Pierangelo Masarati, 2010-04-29.'
~# debuild -us -uc

This build can take a long time − perhaps over an hour − most of which is due to automated testing routines. Eventually, seven new Debian packages will be created in the parent directory, /usr/src/:

ldap-utils_2.4.23-7patched1_i386.deb
libldap-2.4-2-dbg_2.4.23-7patched1_i386.deb
libldap-2.4-2_2.4.23-7patched1_i386.deb
libldap2-dev_2.4.23-7patched1_i386.deb
slapd-dbg_2.4.23-7patched1_i386.deb
slapd-smbk5pwd_2.4.23-7patched1_i386.deb
slapd_2.4.23-7patched1_i386.deb

2. Kerberos client install

First, run the following command to test if the MIT Kerberos V server installed previously is available on the network:

~# nmap -sU -sT -p U:88,464,T:464,749 kdc1.example.com

Starting Nmap 5.00 ( http://nmap.org ) at 2010-12-20 17:13 CET
Interesting ports on kdc1.example.com (192.168.2.36):
PORT    STATE         SERVICE
464/tcp open          kpasswd5
749/tcp open          kerberos-adm
88/udp  open|filtered kerberos-sec
464/udp open|filtered kpasswd5
MAC Address: 08:00:27:AB:13:C1 (Cadmus Computer Systems)

Nmap done: 1 IP address (1 host up) scanned in 1.42 seconds
~# _

If not all of the above ports are shown as open, fix that problem first. If they are, continue by installing these three packages:

~# apt-get install krb5-{config,user} libpam-krb5

A total of three packages are installed as a result with no dependencies:

krb5-config           2.2                            Configuration files for Kerberos Version 5
krb5-user             1.8.3+dfsg-4                   Basic programs to authenticate using MIT Kerberos
libpam-krb5           4.3-1                          PAM module for MIT Kerberos

During the installation, krb5-config will require that a few questions be answered:

Default Kerberos version 5 realm: EXAMPLE.COM
Kerberos servers for your realm: kdc1.example.com kdc2.example.com
Administrative server for your Kerberos realm: krb.example.com

These settings are saved in the Kerberos realm configuration file /etc/krb5.conf. Mostly, it contains information about the realms of a number of other organizations and options regarding Kerberos 4, all of which is unnecessary in this case. The entire contents could be replaced with:

[libdefaults]
        default_realm = EXAMPLE.COM
        forwardable = true
        proxiable = true

[realms]
        EXAMPLE.COM = {
                kdc = kdc1.example.com
                kdc = kdc2.example.com
                admin_server = krb.example.com
        }

See this section for a more detailed explanation of this file.

Regarding the list of KDCs that are specified here, it is often recommended to use a predetermined set of DNS hostname aliases (CNAME records) to refer to the Kerberos servers on a network. However, it is also possible to omit the KDC entries in /etc/krb5.conf and instead rely on DNS SRV resource records to do the same job. See DNS discovery for MIT Kerberos V for information on how to do that.

Use kadmin to create a Kerberos principal for the LDAP service and a matching keytab file by issuing a few commands:

~# kadmin -p admin
Authenticating as principal admin with password.
Password for admin@EXAMPLE.COM: Lampropeltis
kadmin:  addprinc -randkey ldap/ldapks2.example.com
WARNING: no policy specified for ldap/ldapks2.example.com@EXAMPLE.COM; defaulting to no policy
Principal "ldap/ldapks2.example.com@EXAMPLE.COM" created.
kadmin:  ktadd ldap/ldapks2.example.com
Entry for principal ldap/ldapks2.example.com with kvno 2, encryption type AES-256 CTS mode with 96-bit SHA-1 HMAC
added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal ldap/ldapks2.example.com with kvno 2, encryption type ArcFour with HMAC/md5 added to keytab
WRFILE:/etc/krb5.keytab.
Entry for principal ldap/ldapks2.example.com with kvno 2, encryption type Triple DES cbc mode with HMAC/sha1 added
to keytab WRFILE:/etc/krb5.keytab.
Entry for principal ldap/ldapks2.example.com with kvno 2, encryption type DES cbc mode with CRC-32 added to keytab
WRFILE:/etc/krb5.keytab.
kadmin:  q
~# _

The -randkey switch is used to avoid having to use a password. To list the keys in /etc/krb5.keytab, use the klist -ke command. A host (or service) principal and a keytab file should be created for and saved on all of the various client machines that are part of a Kerberos realm.


3. OpenLDAP install

Before starting the install process, run the following command to test if the previously installed OpenLDAP provider server is actually available on the network:

~# nmap -p 389 ldapks1.example.com

Starting Nmap 5.00 ( http://nmap.org ) at 2010-12-20 17:53 CET
Interesting ports on ldapks1.example.com (192.168.2.53):
PORT    STATE SERVICE
389/tcp open  ldap
MAC Address: 08:00:27:43:43:4A (Cadmus Computer Systems)

Nmap done: 1 IP address (1 host up) scanned in 0.23 seconds
~# _

If TCP port 389 is not open, fix that problem first. Otherwise, install three of the newly compiled packages:

~# dpkg -i /usr/src/slapd_2.4.23-7patched1_i386.deb \
/usr/src/ldap-utils_2.4.23-7patched1_i386.deb \
/usr/src/libldap-2.4-2_2.4.23-7patched1_i386.deb

During the install process, an administrator password will be requested for slapd. Use contortrix:

Administrator password: contortrix
Confirm password: contortrix

Run the following command to test if the new OpenLDAP server is actually running:

~# nmap -p 389 localhost

Starting Nmap 5.00 ( http://nmap.org ) at 2010-12-20 17:53 CET
Interesting ports on localhost (127.0.0.1):
PORT    STATE SERVICE
389/tcp open  ldap

Nmap done: 1 IP address (1 host up) scanned in 0.23 seconds
~# _

Perform a quick test by generating an LDIF dump of the contents of a the database:

~# slapcat
hdb_db_open: database "dc=example,dc=com": unclean shutdown detected; 
attempting recovery.
hdb_db_open: database "dc=example,dc=com": recovery skipped in read-only 
mode. Run manual recovery if errors are encountered.
dn: dc=example,dc=com
objectClass: top
objectClass: dcObject
objectClass: organization
o: example.com
dc: example
structuralObjectClass: organization
entryUUID: c868cece-a0ac-102f-8fb8-676e1d316d8b
creatorsName: cn=admin,dc=example,dc=com
createTimestamp: 20101220174613Z
entryCSN: 20101220174613.753133Z#000000#000#000000
modifiersName: cn=admin,dc=example,dc=com
modifyTimestamp: 20101220174613Z

dn: cn=admin,dc=example,dc=com
objectClass: simpleSecurityObject
objectClass: organizationalRole
cn: admin
description: LDAP administrator
userPassword:: e1NTSEF9aWtSc29hMERuUkdFY0EvMi9vTEg2MWZZV25RVlFZMnI=
structuralObjectClass: organizationalRole
entryUUID: c86962f8-a0ac-102f-8fb9-676e1d316d8b
creatorsName: cn=admin,dc=example,dc=com
createTimestamp: 20101220174613Z
entryCSN: 20101220174613.756966Z#000000#000#000000
modifiersName: cn=admin,dc=example,dc=com
modifyTimestamp: 20101220174613Z

~# _

Edit /etc/ldap/ldap.conf and use these two lines:

BASE    dc=example,dc=com
URI     ldap://ldapks2.example.com/

This configuration file is used to set system-wide defaults for LDAP clients.


4. Syncprov

Switch over to the provider server, ldapks1.example.com, to make some changes there. Start by making a number of changes will be made to the LDAP configuration database on ldapks2. Two existing entries will be modified and one new one added. Their current status/absence is as follows:

~# ldapsearch -LLLQY EXTERNAL -H ldapi:/// -b cn=config \
"(|(olcDatabase={1}hdb)(cn=module{0})(olcOverlay={0}syncprov))"
dn: cn=module{0},cn=config
objectClass: olcModuleList
cn: module{0}
olcModulePath: /usr/lib/ldap
olcModuleLoad: {0}back_hdb

dn: olcDatabase={1}hdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcHdbConfig
olcDatabase: {1}hdb
olcDbDirectory: /var/lib/ldap
olcSuffix: dc=example,dc=com
olcAccess: {0}to attrs=userPassword,shadowLastChange by * none
olcAccess: {1}to attrs=loginShell by self write by users read by * none
olcAccess: {2}to dn.base="" by * read
olcAccess: {3}to * by users read by * none
olcLastMod: TRUE
olcRootDN: uid=admin,ou=people,dc=example,dc=com
olcDbCheckpoint: 512 30
olcDbConfig: {0}set_cachesize 0 2097152 0
olcDbConfig: {1}set_lk_max_objects 1500
olcDbConfig: {2}set_lk_max_locks 1500
olcDbConfig: {3}set_lk_max_lockers 1500
olcDbIndex: objectClass eq
olcDbIndex: uid eq
olcDbIndex: cn eq
olcDbIndex: ou eq
olcDbIndex: dc eq

~# _

Here is a description of the changes that will be made to the cn=config DIT:

1.1. With syncrepl replication, the entryUUID attribute is added to all entries in the database. Use of the olcSpSessionlog option (see description below) requires searching for this attribute, so it is recommended (but not required) to set an equality index for it. Doing so on the provider server can improve session log performance considerably.
1.2. Similarly, it is recommended to add an eq index for the entryCSN attribute when making use of the olcSpCheckpoint option.
2. For the cn=module{0} entry, an extra olcModuleLoad attribute will be added to load the syncprov module into the slapd process. This is because it was not compiled statically into /usr/sbin/slapd.
3. A new entry, olcOverlay={0}syncprov, will be added to the cn=config DIT to activate and configure the syncprov module.

To make both of the above changes to the cn=config DIT, first create an LDIF file, called ~/olc-mod3.ldif, with the following contents:

# 1.1.
dn: olcDatabase={1}hdb,cn=config
changetype: modify
add: olcDbIndex
olcDbIndex: entryUUID eq
-
# 1.2.
add: olcDbIndex
olcDbIndex: entryCSN eq

# 2.
dn: cn=module{0},cn=config
changetype: modify
add: olcModuleLoad
olcModuleLoad: syncprov

# 3.
dn: olcOverlay=syncprov,olcDatabase={1}hdb,cn=config
changetype: add
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: {0}syncprov
olcSpCheckpoint: 100 10
olcSpSessionlog: 100

See this section for an explanation of the last two modifications.

Apply the changes in the LDIF file with this command:

~# ldapmodify -QY EXTERNAL -H ldapi:/// -f ~/olc-mod3.ldif
modifying entry "olcDatabase={1}hdb,cn=config"

modifying entry "cn=module{0},cn=config"

adding new entry "olcOverlay=syncprov,olcDatabase={1}hdb,cn=config"

~# _

Rerun the previous ldapsearch command to verify that all of the changes have been made successfully. Note that an index number has been added automatically to the RDN for the syncprov overlay.


5. Kstart

Return to ldapks2. To keep its copy of the database in sync, the consumer must contact the provider and authenticate itself just like any other client. It has previously been shown how to do this with a simple bind and a password, but in this case Kerberos will be used to take care of both slapd consumer authentication to the provider and encryption of the entire replication process. To make this possible, an initial Kerberos ticket (a TGT) must be obtained automatically on behalf of the openldap user, which owns the slapd process, and it must be renewed regularly. One way to do this is to run kinit -k with a cron job, but a better solution is to use a modified version of kinit, called k5start. Install it with:

~# apt-get install kstart

This is the only package that gets installed as a result:

kstart               3.16-3                  Kerberos kinit supporting AFS and ticket refreshing

To configure it, just add this line to the end of the /etc/inittab file to start running k5start in the background soon after the system boots up:

KS:2345:respawn:/usr/bin/k5start -U -f /etc/krb5.keytab -K 10 -l 24h -k /tmp/krb5cc_105 -o openldap

A number of options have been used for this command:

-U Determine the principal to authenticate based on the first entry in the Kerberos keytab file. Must be used with the -f option.
-f /etc/krb5.keytab Specifies the full path of the Kerberos keytab file.
-K 10 Reawaken the daemon every 10 minutes to check if the ticket needs to be renewed.
-l 24h Set the ticket lifetime to 24 hours (to match the actual ticket lifetime used in this example).
-k /tmp/krb5cc_105 Use the file /tmp/krb5cc_105 as the ticket cache. The number at the end of the file name *must match* the UID (105 in this example) of the user on behalf of whom the ticket is maintained (see option -o). Otherwise this slapd server will not authenticate properly and the result will be a ldap_sasl_interactive_bind_s failed (-2) error.
-o openldap The name of the user account that is to become the owner of the ticket cache file (see option -k).

After saving this modification to /etc/inittab, start k5start for the first time by forcing init to reload its configuration file:

~# kill -HUP 1

A new ticket cache file, /tmp/krb5cc_105, should be created almost immediately as a consequence.


6. Slapd kerberization

This step is about kerberizing OpenLDAP. A number of requirements have already been met, but there is more. First, install this one package:

~# apt-get install libsasl2-modules-gssapi-mit

Only one package is installed as a result with no dependencies:

libsasl2-modules-gssapi-mit      2.1.23.dfsg1-6              Cyrus SASL - pluggable authentication modules (GSSAPI)

First, change the permissions and ownership of the Kerberos service keytab file to allow slapd to read it:

~# chmod 640 /etc/krb5.keytab
~# chown root.openldap /etc/krb5.keytab
~# _

Next, three different files must be modified. First, edit /etc/default/slapd and uncomment a line near the end that exports as a variable the location of the Kerberos system keytab file:

export KRB5_KTNAME=/etc/krb5.keytab

Edit /etc/ldap/ldap.conf and add the following line, which specifies the authentication mechanism, to the end of the file:

SASL_MECH GSSAPI

The last part of this kerberization step is, just as on ldapks1, to get slapd to map the GSSAPI-format user names to LDAP names. To do this, only the root object of the cn=config DIT needs to be modified. Currently, it looks like this:

~# ldapsearch -LLLQY EXTERNAL -H ldapi:/// -b cn=config cn=config
dn: cn=config
objectClass: olcGlobal
cn: config
olcArgsFile: /var/run/slapd/slapd.args
olcLogLevel: none
olcPidFile: /var/run/slapd/slapd.pid
olcToolThreads: 1

~# _

Here is a description of the changes that will be made to ldaps2's runtime configuration:

1.1. For the root object, cn=config, the value of the olcLogLevel attribute will be changed from none to stats. This will cause slapd to log more about what it does to /var/log/syslog. It is not related to this kerberization task, but it is easy to do it now. It is better for testing purposes, although its current setting (none) is better for production environments.
1.2. For the root object, cn=config, add an olcAuthzRegexp attribute, the value of which consists of a match string and a replace string. The match string will be a regular expression to match the simple user names provided by the SASL subsystem. The replace string will then convert them to an LDAP DN format.
1.3. An olcSaslRealm attribute will be added to specify the SASL realm, in this case a Kerberos realm.

To make the above changes to the slapd runtime configuration on ldapks2, create an LDIF file, called ~/olc-mod1.ldif, with the following contents:

# 1.1.
dn: cn=config
changetype: modify
replace: olcLogLevel
olcLogLevel: stats
-
# 1.2.
add: olcAuthzRegexp
olcAuthzRegexp: uid=([^,]+),cn=example.com,cn=gssapi,cn=auth
  uid=$1,ou=people,dc=example,dc=com
-
# 1.3.
add: olcSaslRealm
olcSaslRealm: EXAMPLE.COM

Th authz-regexp statement used here is exactly the same as on the provider server, ldapks1.

Apply the changes in the LDIF file with this command:

~# ldapmodify -QY EXTERNAL -H ldapi:/// -f ~/olc-mod1.ldif
modifying entry "cn=config"

~# _

Rerun the previous ldapsearch command to verify that all of the changes have been made successfully.

Because of the new software that has been added and the changes that have also been made outside of the slapd server, it must be restarted before they will all have the desired effect:

~# /etc/init.d/slapd restart
Stopping OpenLDAP: slapd.
Starting OpenLDAP: slapd.
~# _

7. Syncrepl

The next set of changes to be made to the slapd runtime configuration on ldaps2 will enable the replication process for this system as a slapd consumer. For that, only the database definition object needs to be changed, which currently looks like this:

~# ldapsearch -LLLQY EXTERNAL -H ldapi:/// -b cn=config olcDatabase={1}hdb
dn: olcDatabase={1}hdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcHdbConfig
olcDatabase: {1}hdb
olcDbDirectory: /var/lib/ldap
olcSuffix: dc=example,dc=com
olcAccess: {0}to attrs=userPassword,shadowLastChange by self write by anonymou
 s auth by dn="cn=admin,dc=example,dc=com" write by * none
olcAccess: {1}to dn.base="" by * read
olcAccess: {2}to * by self write by dn="cn=admin,dc=example,dc=com" write by *
  read
olcLastMod: TRUE
olcRootDN: cn=admin,dc=example,dc=com
olcRootPW: {SSHA}ikRsoa0DnRGEcA/2/oLH61fYWnQVQY2r
olcDbCheckpoint: 512 30
olcDbConfig: {0}set_cachesize 0 2097152 0
olcDbConfig: {1}set_lk_max_objects 1500
olcDbConfig: {2}set_lk_max_locks 1500
olcDbConfig: {3}set_lk_max_lockers 1500
olcDbIndex: objectClass eq

~# _

Here is a description of the changes that will be made to the database definition object on ldaps2:

1.1.x. All three of the current ACLs, each one a value of an olcAccess attribute, will be deleted. For the sake of clarity, this will be done in reverse order, or else the index numbers used to match them would all have to be zero.
1.2.1. The first new olcAccess attribute (index 0) will prevent any access to password information stored in the DIT.
1.2.2. The third ACL (index 2) is will allow read access to the base of the tree for things like supportedSASLMechanisms.
1.2.3. The fourth and last ACL (index 3) will allow only authenticated users to have read access to everything else in the LDAP database. Anonymous users will have no access.
1.3. The value for the olcRootDN attribute, which indicates the administrative account for the database, will be changed to something completely arbitrary. That is because there is no need for it to be part of the directory. However, setting a value for this attribute is still mandatory for an OpenLDAP consumer in order to allow it to store anything in the database that it receives, regardless of how the ACLs are set.
1.4. Since the administrative account is no longer part of the dc=example,dc=com DIT, the olcRootPW attribute will be deleted.
1.5. An eq index will be set for the entryCSN attribute to speed up the replication process.
1.6. Idem for the entryUUID attribute.
1.7. An equality (eq) index will be set for the uid attribute to speed up the authentication process. It will also prevent "bdb_equality_candidates" errors...
1.8. An equality index will also be set for Common Name (cn) objects to prevent "bdb_equality_candidates" errors from appearing in the log file, /var/log/syslog. One such error will otherwise appear in the log for every search for an entry of this type.
1.9. Idem for Organizational Unit (ou) objects.
1.10. Idem for Domain Component (dc) objects.
1.11. The olcSyncrepl attribute will be added to configure LDAP Sync Replication engine. The options used for it are explained below.

To make the above changes to the slapd runtime configuration on ldapks2, create an LDIF file, called ~/olc-mod2.ldif, with the following contents:

# 1.1.1.
dn: olcDatabase={1}hdb,cn=config
changetype: modify
delete: olcAccess
olcAccess: {2}to *
  by self write
  by dn="cn=admin,dc=example,dc=com" write
  by * read
-
# 1.1.2.
delete: olcAccess
olcAccess: {1}to dn.base=""
  by * read
-
# 1.1.3.
delete: olcAccess
olcAccess: {0}to attrs=userPassword,shadowLastChange
  by self write
  by anonymous auth
  by dn="cn=admin,dc=example,dc=com" write
  by * none
-
# 1.2.1.
add: olcAccess
olcAccess: {0}to attrs=userPassword,shadowLastChange
  by * none
-
# 1.2.2.
add: olcAccess
olcAccess: {2}to dn.base=""
  by * read
-
# 1.2.3.
add: olcAccess
olcAccess: {3}to *
  by users read
  by * none
-
# 1.3.
replace: olcRootDN
olcRootDN: cn=manager
-
# 1.4.
delete: olcRootPW
-
# 1.5.
add: olcDbIndex
olcDbIndex: entryCSN eq
-
# 1.6.
add: olcDbIndex
olcDbIndex: entryUUID eq
-
# 1.7.
add: olcDbIndex
olcDbIndex: uid eq
-
# 1.8.
add: olcDbIndex
olcDbIndex: cn eq
-
# 1.9.
add: olcDbIndex
olcDbIndex: ou eq
-
# 1.10.
add: olcDbIndex
olcDbIndex: dc eq
-
# 1.11.
add: olcSyncrepl
olcSyncrepl: rid=123
  provider="ldap://ldapks.example.com:389/"
  type=refreshAndPersist
  retry="60 30 300 +"
  searchbase="dc=example,dc=com"
  bindmethod=sasl
  saslmech=gssapi

The presence of the olcSyncrepl attribute will indicate that the current database is a replica. This directive has many different options, a number of which are used here:

rid The replica ID of the current syncrepl directive on the consumer. This must be a non-negative three-digit integer. The choice here, 123, is arbitrary.
provider The name of the provider server with master content to synchrinize with. The name is written as an LDAP URI followed by a port number if necessary. Here, the alias is used that refers to the current provider server, ldapks1.
type The operation type. This can be either refreshOnly, which is polling pull-based synchronization, or refreshAndPersist, which is persistent push-based synchronization. If the latter type is used, the retry option is required.
retry In case of replication errors, this option determines both the amount of time in seconds to wait before the consumer attempts to reconnect to the provider, as well as the number of retries. Specified as a set of one or more interval-retry pairs, separated by spaces. A "+" indicates an indefinite number of retries. The setting used here means that, if a replication error occurs, the consumer will attempt to reconnect with the provider once every 60 seconds with 30 retries, and thereafter once every 300 seconds with an indefinite (+) number of retries.
searchbase The search base to use for replication, written as a DN. This option has no default and must always be specified.
bindmethod The security method to be used for authentication. This can be either simple or sasl. Since the latter method are used here, the option saslmech is required.
saslmech This option, for SASL mechanisms, can have any of a number of different values, depending on the authentication mechanism used. Here, gssapi is specified.

The above syncrepl configuration uses persistent push-based synchronization in which the provider keeps track of the consumer, sending it the necessary updates whenever its own database is modified. This is often desirable when modifications are frequent, but if not, use the refreshOnly operation type instead:

# 2.11. (alternative)
add: olcSyncrepl
olcSyncrepl: rid=123
  provider="ldap://ldapks.example.com:389/"
  type=refreshOnly
  interval=00:00:05:00
  searchbase="dc=example,dc=com"
  bindmethod=sasl
  saslmech=gssapi
interval The replication interval specified in dd:hh:mm:ss. Limited to use with refreshOnly synchronization. The default is one day.

This alternative configuration features the polling, pull-based refreshOnly operation type. The interval setting used here, 00:00:05:00, means that replication will occur once every five minutes.

Anyway, apply the changes in the LDIF file with this command:

~# ldapmodify -QY EXTERNAL -H ldapi:/// -f ~/olc-mod2.ldif
modifying entry "olcDatabase={1}hdb,cn=config"

~# _

Rerun the previous ldapsearch command to verify that all of the changes have been made successfully.

At this point the consumer will already have synchronized its database with the provider. A quick slapcat will confirm this. However, the two copies of the database are not really in sync: the root object is still the same and the default cn=admin object is still present. To fix this situation, the consumer's database must resynchronized with the provider:

~# /etc/init.d/slapd stop
Stopping OpenLDAP: slapd.
~# rm /var/lib/ldap/*
~# /etc/init.d/slapd start
Starting OpenLDAP: slapd.
~# _

This will stop the LDAP service on ldapks2, delete its copy of the database and restart the service, at which point it will fetch a fresh copy from the provider. Run slapcat once more to see the difference.


8. Referrals

LDAP Sync Replication does present one important problem for LDAP clients: the database containing the DIT on a consumer server is always read-only. To allow clients to make modifications to the DIT anyway, the consumer must either refer its clients on to the provider, or proxy their requests. Both of these options will be arranged by modifying two existing entries in the consumer server's configuration DIT and adding two new ones. The following search command on ldapks2 will verify their current status/absence:

~# ldapsearch -LLLQY EXTERNAL -H ldapi:/// -b cn=config "(|(cn=module{0})\
(olcDatabase={1}hdb)(olcOverlay={0}chain)(olcDatabase={0}ldap))"
dn: cn=module{0},cn=config
objectClass: olcModuleList
cn: module{0}
olcModulePath: /usr/lib/ldap
olcModuleLoad: {0}back_hdb

dn: olcDatabase={1}hdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcHdbConfig
olcDatabase: {1}hdb
olcDbDirectory: /var/lib/ldap
olcSuffix: dc=example,dc=com
olcAccess: {0}to attrs=userPassword,shadowLastChange by * none
olcAccess: {1}to dn.base="" by * read
olcAccess: {2}to * by users read by * none
olcLastMod: TRUE
olcRootDN: cn=manager
olcSyncrepl: {0}rid=123 provider="ldap://ldapks.example.com:389/" type=refresh
 AndPersist retry="60 30 300 +" searchbase="dc=example,dc=com" bindmethod=sasl
  saslmech=gssapi
olcDbCheckpoint: 512 30
olcDbConfig: {0}set_cachesize 0 2097152 0
olcDbConfig: {1}set_lk_max_objects 1500
olcDbConfig: {2}set_lk_max_locks 1500
olcDbConfig: {3}set_lk_max_lockers 1500
olcDbIndex: objectClass eq
olcDbIndex: entryCSN eq
olcDbIndex: entryUUID eq
olcDbIndex: uid eq
olcDbIndex: cn eq
olcDbIndex: ou eq
olcDbIndex: dc eq

~# _

Here is a description of the changes that will be made to the cn=config DIT on ldapks2:

1. For the olcDatabase={1}hdb entry, the olcUpdateref attribute will be added with a URL for the provider server. This referral will then be passed directly on to clients whenever they ask slapd to modify the read-only replica, e.g. to change their shell type. Unfortunately, although this is the preferred method, not all LDAP clients understand referrals. Therefore, it is a good idea to also configure the chain overlay, which allows automatic referral chasing, i.e. for the consumer to act as a proxy.
2. For the cn=module{0} entry, an additional olcModuleLoad attribute will be added to load the back_ldap (LDAP backend database) module into the slapd process. This will make the chain overlay available, which happens to be built into the back_ldap module.
3. A new entry, olcOverlay={0}chain, will be added to activate and configure the chain overlay. This will allow slapd to act as a proxy, automatically chasing referrals on behalf of clients. Although this overlay has very few directives of its own, those belonging to the LDAP backend assume a special meaning when used in conjunction with it.
4. Another new entry, olcDatabase={0}ldap, will be added to configure the LDAP backend database. This is not an actual database, but provides basic proxying functionality, in this case for the chain overlay, including configuration directives necessary for it to contact the provider server. Some of its details are explained below.

To make the above changes to the cn=config DIT on ldapks2, create an LDIF file, called ~/olc-mod3.ldif, with the following contents:

# 1.
dn: olcDatabase={1}hdb,cn=config
changetype: modify
add: olcUpdateref
olcUpdateref: "ldap://ldapks.example.com:389/"

# 2.
dn: cn=module{0},cn=config
changetype: modify
add: olcModuleLoad
olcModuleLoad: {1}back_ldap

# 3.
dn: olcOverlay=chain,olcDatabase={-1}frontend,cn=config
changetype: add
objectClass: olcOverlayConfig
objectClass: olcChainConfig
olcOverlay: {0}chain
olcChainReturnError: TRUE

# 4.
dn: olcDatabase=ldap,olcOverlay={0}chain,olcDatabase={-1}frontend,
 cn=config
changetype: add
objectClass: olcLDAPConfig
objectClass: olcChainDatabase
olcDatabase: {0}ldap
olcDbURI: "ldap://ldapks.example.com:389/"
olcDbRebindAsUser: TRUE
olcDbIDAssertBind: bindmethod=sasl
  saslmech=gssapi
  mode=self

Here, the olcDbIDAssertBind directive defines the parameters of the authentication method used by ldapks2 to authorize connections from authenticated users. It has a number of different options, a few of which are used here:

bindmethod The method with which the consumer server authenticates to the provider. Here, the sasl authentication method is used, in which case the saslmech attribute is also required.
saslmech This option, for SASL mechanisms, can have any of a number of different values, depending on the authentication mechanism used. Here, gssapi is specified.
mode When the mode self is used, the client's identity will be asserted when a request is proxied.

Apply the above changes on ldapks2 with this command:

~# ldapmodify -QY EXTERNAL -H ldapi:/// -f ~/olc-mod3.ldif
modifying entry "olcDatabase={1}hdb,cn=config"

modifying entry "cn=module{0},cn=config"

adding new entry "olcOverlay=chain,olcDatabase={1}hdb,cn=config"

adding new entry "olcDatabase=ldap,olcOverlay={0}chain,olcDatabase={1}hdb,cn=config"

~# _

Rerun the previous ldapsearch command to verify that all of the changes have been made successfully.


9. Authentication test

Run some tests on ldapks2. First try a simple unauthenticated (-x) LDAP query:

~# ldapsearch -x -LLL
No such object (32)
~# _

This should not work, because modification 1.2.4 in step 9 prevents unauthenticated users from reading the database. However, the authenticated version of this query will currently give an error:

~# ldapsearch -LLL
SASL/GSSAPI authentication started
ldap_sasl_interactive_bind_s: Local error (-2)
	additional info: SASL(-1): generic failure: GSSAPI Error: 
Unspecified GSS failure.  Minor code may provide more information 
(Credentials cache file '/tmp/krb5cc_0' not found)
~# _

The solution is to first acquire a Kerberos ticket for the admin user (password Lampropeltis):

~# kinit admin
Password for admin@EXAMPLE.COM: Lampropeltis
~# _

A verification of the ticket should show a success:

~# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: admin@EXAMPLE.COM

Valid starting     Expires            Service principal
12/22/10 21:08:32  12/23/10 21:08:26  krbtgt/EXAMPLE.COM@EXAMPLE.COM
12/22/10 21:08:37  12/23/10 21:08:26  ldap/ldapks2.example.com@EXAMPLE.COM
~# _

Now the authenticated version of the query should work:

~# ldapsearch -LLL
SASL/GSSAPI authentication started
SASL username: admin@EXAMPLE.COM
SASL SSF: 56
SASL data security layer installed.
dn: dc=example,dc=com
objectClass: top
objectClass: dcObject
objectClass: organization
o: example.com
dc: example

dn: ou=people,dc=example,dc=com
objectClass: organizationalUnit
ou: people

dn: ou=groups,dc=example,dc=com
objectClass: organizationalUnit
ou: groups

~# _

10. Proxy authorization

Before the chaining configuration added in step 10 will work, more changes first have to be made to the OpenLDAP provider server, ldapks1. What is required is something called SASL proxy authorization. With this mechanism it is possible to allow an authenticated user, such as the consumer server in this example, to assume the identity of other users and use that to relay their write requests on to the provider server. The ability to assume another user's identity is important, since it would otherwise not be possible to enforce existing access rules.

Two changes are needed to enable proxy authorization. First, a new LDAP object must be created that corresponds to the consumer user's GSSAPI (Kerberos) identity and includes an authzTo attribute that will allow it to act as an authorization proxy for certain users. A new organizational unit will also be created in which the new consumer server object will reside. Second, the provider server, ldapks1, will be configured to accept such proxy requests.

The new LDAP object to represent ldapks2 is required because of the need to authorize it to act as a proxy entity. At the moment it is only able to replicate with ldapks1 thanks to Kerberos authentication, but without an LDAP object to represent it as well, any form of LDAP authorization will not be possible. An examination of /var/log/syslog on ldapks1 will show that, when ldapks2 binds for the purpose of replication, its LDAP identity is currently part of ou=people:

BIND authcid="ldap/ldapks2.example.com" \
     authzid="ldap/ldapks2.example.com"
BIND dn="uid=ldap/ldapks2.example.com,ou=people,dc=example,dc=com" \
     mech=GSSAPI sasl_ssf=56 ssf=56

This must be changed. On ldapks1, start by creating an LDIF file, called ~/ldapks2.ldif, with the following contents:

dn: ou=consumers,dc=example,dc=com
objectClass: organizationalUnit
ou: consumers

dn: cn=ldapks2,ou=consumers,dc=example,dc=com
cn: ldapks2
objectClass: simpleSecurityObject
objectClass: organizationalRole
description: LDAP server2 replicator
authzTo: dn.regex:^uid=[^,]+,ou=people,dc=example,dc=com$
userPassword: {CRYPT}*

This will create and origanizational unit and an organizationalRole object. Note the last two attributes: authzTo is a source rule that determines which user identities it is allowed to assume; in this case its value is a regular expression that matches all uid objects in the ou=people,dc=example,dc=com container. The userPassword attribute, on the other hand, is only included because it is required for organizationalRole objects, but since it is not actually necessary here, an invalid hash is used for its value.

To add these new objects, authenticate as the administrative user and add them to the database using the following commands:

root@ldapks1:~# kinit admin
Password for admin@EXAMPLE.COM: Lampropeltis
root@ldapks1:~# ldapadd -f ~/ldapks2.ldif
SASL/GSSAPI authentication started
SASL username: admin@EXAMPLE.COM
SASL SSF: 56
SASL data security layer installed.
adding new entry "ou=consumers,dc=example,dc=com"

adding new entry "cn=ldapks2,ou=consumers,dc=example,dc=com"

root@ldapks1:~# _

Now to make the second of the two changes. To ensure that the provider server, ldapks1, will accept proxy requests associated with the authzTo attribute, its cn=config DIT must be modified. This involves two entries that currently look like this:

~# ldapsearch -LLLQY EXTERNAL -H ldapi:/// -b cn=config \
"(|(cn=config)(olcDatabase={1}hdb))"
dn: cn=config
objectClass: olcGlobal
cn: config
olcArgsFile: /var/run/slapd/slapd.args
olcAuthzRegexp: {0}uid=([^,]+),cn=example.com,cn=gssapi,cn=auth uid=$1,ou=peop
 le,dc=example,dc=com
olcLogLevel: stats
olcPidFile: /var/run/slapd/slapd.pid
olcSaslRealm: EXAMPLE.COM
olcToolThreads: 1

dn: olcDatabase={1}hdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcHdbConfig
olcDatabase: {1}hdb
olcDbDirectory: /var/lib/ldap
olcSuffix: dc=example,dc=com
olcAccess: {0}to attrs=userPassword,shadowLastChange by * none
olcAccess: {1}to attrs=loginShell by self write by users read by * none
olcAccess: {2}to dn.base="" by * read
olcAccess: {3}to * by users read by * none
olcLastMod: TRUE
olcRootDN: uid=admin,ou=people,dc=example,dc=com
olcDbCheckpoint: 512 30
olcDbConfig: {0}set_cachesize 0 2097152 0
olcDbConfig: {1}set_lk_max_objects 1500
olcDbConfig: {2}set_lk_max_locks 1500
olcDbConfig: {3}set_lk_max_lockers 1500
olcDbIndex: objectClass eq
olcDbIndex: uid eq
olcDbIndex: cn eq
olcDbIndex: ou eq
olcDbIndex: dc eq
olcDbIndex: entryUUID eq
olcDbIndex: entryCSN eq

~# _

Here is a description of the changes that will be made to the cn=config tree on ldapks1 to enable proxy authorization:

1.1.1. For the root object, cn=config, an olcAuthzRegexp statement must be added to only match GSSAPI-format names involving Kerberos service principals for the LDAP protocol. Since this olcAuthzRegexp statement must come before the catch-all statement for ou=people, the existing one must first be deleted.
1.1.2. Add an olcAuthzRegexp statement with index 0 to match only GSSAPI-format names involving Kerberos service principals for the LDAP protocol.
1.1.3. Recreate the catch-all olcAuthzRegexp statement for ou=people, but now with index 1.
1.2. In addition, a, authz-policy attribute will be added to enable proxy authorization using rules associated with the authzTo attribute of the authentication DN (in this case cn=ldapks2). Without this, the authzTo attribute that was added to the cn=ldapks2 object would be ignored.
2.1.1. For the olcDatabase={1}hdb entry, the first olcAccess attribute (with index 0) will be modified to ensure that all entries in ou=consumers, including ldapks2, will have full access to the DIT. However, since this is a multivalue attribute, it must first be deleted.
2.1.2. When the first olcAccess attribute (with index 0) is recreated, the ACL for it will include read access for all entries in ou=consumers.

To make the above changes in the same order to the cn=config DIT, create an LDIF file on ldapks1, called ~/olc-mod4.ldif, with the following contents:

# 1.1.1.
dn: cn=config
changetype: modify
delete: olcAuthzRegexp
olcAuthzRegexp: {0}uid=([^,]+),cn=example.com,cn=gssapi,cn=auth
  uid=$1,ou=people,dc=example,dc=com
-
# 1.1.2.
add: olcAuthzRegexp
olcAuthzRegexp: {0}uid=ldap/([^/\.]+).example.com,cn=example.com,
 cn=gssapi,cn=auth cn=$1,ou=consumers,dc=example,dc=com
-
# 1.1.3.
add: olcAuthzRegexp
olcAuthzRegexp: {1}uid=([^,]+),cn=example.com,cn=gssapi,cn=auth
  uid=$1,ou=people,dc=example,dc=com
-
# 1.2.
add: olcAuthzPolicy
olcAuthzPolicy: to

# 2.1.1.
dn: olcDatabase={1}hdb,cn=config
changetype: modify
delete: olcAccess
olcAccess: {0}to attrs=userPassword,shadowLastChange
  by * none
-
# 2.1.2.
add: olcAccess
olcAccess: {0}to attrs=userPassword,shadowLastChange
  by dn.one="ou=consumers,dc=example,dc=com" read
  by * none

Run this command on ldapks1 to apply the change:

~# ldapmodify -QY EXTERNAL -H ldapi:/// -f ~/olc-mod4.ldif
modifying entry "cn=config"

modifying entry "olcDatabase={1}hdb,cn=config"

~# _

Rerun the previous search command to verify that the changes have been made correctly.

For some reason, in this particular situation, despite the runtime configuration, slapd still has to be restarted before proxy authorization will work:

~# /etc/init.d/slapd restart
Starting OpenLDAP: slapd.
~# _

11. Proxy auth test

This last step will test whether the newly added proxy authorization actually works. This is easily done on the consumer server, ldapks2, as the admin user, using the ldapadd command. If it works, the newly added entries will become immediately available on the consumer server (assuming refreshAndPersist replication is used). First, on ldapks2, create a file, called ~/bbeamon.ldif to add a new user account based on an existing Kerberos account by the same name, and add this information to it:

dn: cn=bbeamon,ou=groups,dc=example,dc=com
cn: bbeamon
gidNumber: 20002
objectClass: top
objectClass: posixGroup

dn: uid=bbeamon,ou=people,dc=example,dc=com
uid: bbeamon
uidNumber: 20002
gidNumber: 20002
cn: Bob
sn: Beamon
objectClass: top
objectClass: person
objectClass: posixAccount
objectClass: shadowAccount
loginShell: /bin/bash
homeDirectory: /home/bbeamon
userPassword: {CRYPT}*

Authenticate first if necessary:

~# kinit admin
Password for admin@EXAMPLE.COM: Lampropeltis
~# _

Then add the new user:

~# ldapadd -f bbeamon.ldif
SASL/GSSAPI authentication started
SASL username: admin@EXAMPLE.COM
SASL SSF: 56
SASL data security layer installed.
adding new entry "cn=bbeamon,ou=groups,dc=example,dc=com"

adding new entry "uid=bbeamon,ou=people,dc=example,dc=com"

~# _

If that works, the following entries should be seen in the /var/log/syslog file on ldapks1 as evidence of the proxy event that just transpired:

conn=1000 fd=15 ACCEPT from IP=192.168.2.56:41914 (IP=0.0.0.0:389)
conn=1000 op=0 BIND dn="" method=163
conn=1000 op=0 RESULT tag=97 err=14 text=SASL(0): successful result: 
conn=1000 op=1 BIND dn="" method=163
conn=1000 op=1 RESULT tag=97 err=14 text=SASL(0): successful result: 
conn=1000 op=2 BIND dn="" method=163
conn=1000 op=2 BIND authcid="ldap/ldapks2.example.com@EXAMPLE.COM" authzid="ldap/ldapks2.example.com@EXAMPLE.COM"
conn=1000 op=2 BIND dn="cn=ldapks2,ou=consumers,dc=example,dc=com" mech=GSSAPI sasl_ssf=56 ssf=56
conn=1000 op=2 RESULT tag=97 err=0 text=
connection_input: conn=1000 deferring operation: binding
conn=1000 op=3 PROXYAUTHZ dn="uid=admin,ou=people,dc=example,dc=com"
conn=1000 op=3 ADD dn="cn=bbeamon,ou=groups,dc=example,dc=com"
conn=1000 op=3 RESULT tag=105 err=0 text=
conn=1001 fd=22 ACCEPT from IP=192.168.2.56:41915 (IP=0.0.0.0:389)
conn=1001 op=0 BIND dn="" method=163
conn=1001 op=0 RESULT tag=97 err=14 text=SASL(0): successful result: 
conn=1001 op=1 BIND dn="" method=163
conn=1001 op=1 RESULT tag=97 err=14 text=SASL(0): successful result: 
conn=1001 op=2 BIND dn="" method=163
conn=1001 op=2 BIND authcid="ldap/ldapks2.example.com@EXAMPLE.COM" authzid="ldap/ldapks2.example.com@EXAMPLE.COM"
conn=1001 op=2 BIND dn="cn=ldapks2,ou=consumers,dc=example,dc=com" mech=GSSAPI sasl_ssf=56 ssf=56
conn=1001 op=2 RESULT tag=97 err=0 text=
connection_input: conn=1001 deferring operation: binding
conn=1001 op=3 PROXYAUTHZ dn="uid=admin,ou=people,dc=example,dc=com"
conn=1001 op=3 ADD dn="uid=bbeamon,ou=people,dc=example,dc=com"
conn=1001 op=3 RESULT tag=105 err=0 text=

Further examination of the syslog on ldapks1 should also show that whenever ldapks2 is restarted, its replication process binds to the provider with the new LDAP name created for it in step 12:

conn=1003 fd=15 ACCEPT from IP=192.168.2.56:34012 (IP=0.0.0.0:389)
conn=1003 op=0 BIND dn="" method=163
conn=1003 op=0 RESULT tag=97 err=14 text=SASL(0): successful result: 
conn=1003 op=1 BIND dn="" method=163
conn=1003 op=1 RESULT tag=97 err=14 text=SASL(0): successful result: 
conn=1003 op=2 BIND dn="" method=163
conn=1003 op=2 BIND authcid="ldap/ldapks2.example.com@EXAMPLE.COM" authzid="ldap/ldapks2.example.com@EXAMPLE.COM"
conn=1003 op=2 BIND dn="cn=ldapks2,ou=consumers,dc=example,dc=com" mech=GSSAPI sasl_ssf=56 ssf=56
conn=1003 op=2 RESULT tag=97 err=0 text=
conn=1003 op=3 SRCH base="dc=example,dc=com" scope=2 deref=0 filter="(objectClass=*)"
conn=1003 op=3 SRCH attr=* +

12. Bug warning

Unfortunately, there is another bug in the current version of slapd for which there is, as yet, no patch. It only affects proxy authorization when using SASL binds with the GSSAPI mechanism − replication is unaffected. With version 2.4.23-7, it may be possible to install the provider and a consumer and complete the above test successfully, but this good behavior is likely to cease later on, perhaps after restarting one or both systems.

The problem is that, when the consumer authenticates to the provider, it inexplicably does so with a SIMPLE bind (method=128), instead of with SASL-GSSAPI (method=163). Consequently, no new objects can be added to the DIT from the consumer and the client returns with an error:

~# ldapadd -Qf bbeamon.ldif
adding new entry "cn=bbeamon,ou=groups,dc=example,dc=com"
ldap_add: Strong(er) authentication required (8)

~# _

Hopefully, a fix will become available soon. When it does, the instructions for applying it will be included here.


13. See also
14. Further reading
  • Eastlake D, Panitz A. 1999. RFC2606 − Reserved Top Level DNS Names. The Internet Society. HTML at the Internet FAQ Archives.
  • Hodges J, Morgan R. 2002. RFC3377 − Lightweight Directory Access Protocol (v3): Technical Specification. The Internet Society. HTML at the Internet FAQ Archives.
  • Kohl J, Neuman C. 1993. RFC1510 − The Kerberos Network Authentication Service (V5). HTML at the Internet FAQ Archives.
  • Linn J. 2000. RFC2743 − Generic Security Service Application Program Interface Version 2, Update 1. The Internet Society. HTML at the Internet FAQ Archives.
  • Myers J. 1997. RFC2222 − Simple Authentication and Security Layer (SASL). The Internet Society. HTML at the Internet FAQ Archives.
  • Wahl M, Howes T, Kille S. 1997. RFC2251 − Lightweight Directory Access Protocol (v3). The Internet Society. HTML at the Internet FAQ Archives.
  • Wray J. 2000. RFC2744 − Generic Security Service API Version 2 : C-bindings. The Internet Society. HTML at the Internet FAQ Archives.
  • Yeong W, Howes T, Kille S. 1993. RFC1487 − X.500 Lightweight Directory Access Protocol. The Internet Society. HTML at the Internet FAQ Archives.

15. Sources
  • Carter G. 2003. LDAP System Administration. O'Reilly & Associates, Inc. ISBN 1-56592-491-6. 294 pp.
  • Massachusetts Institute of Technology. 1985-2010. Kerberos V5 System Administrator's Guide. HTML at the Massachusetts Institute of Technology (MIT).
  • Milicchio F, Gehrke WA. 2007. Distributed Services with OpenAFS. Springer-Verlag. ISBN-13 978-3-540-36633-1. 395 pp.
  • OpenLDAP Project. 2009. OpenLDAP Software 2.4 Administrator's Guide. HTML at OpenLDAP.


Last modified: 2012-08-24, 17:19

©2003-2014 RJ Systems. Permission is granted to copy, distribute and/or modify the
content of this page under the terms of the OpenContent License, version 1.0.