ipa-server-upgrade: IPv6 stack is enabled in the kernel but there is no interface that has ::1 address assigned

I applied the latest CentOS updates, as usual. It included a kernel update, so I rebooted the system:

$ sudo yum update -y
$ sudo reboot

After reboot, ipactl showed that FreeIPA was not running:

$ sudo ipactl status
Directory Service: STOPPED
Directory Service must be running in order to obtain status of other services
ipa: INFO: The ipactl command was successful

I tried to start it:

$ sudo ipactl start
Upgrade required: please run ipa-server-upgrade command
Aborting ipactl

I tried running ipa-server-upgrade:

$ sudo ipa-server-upgrade
IPv6 stack is enabled in the kernel but there is no interface that has ::1 address assigned. Add ::1 address resolution to 'lo' interface. You might need to enable IPv6 on the interface 'lo' in sysctl.conf.
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

I had previously disabled IPv6 in /etc/sysctl.conf and removed the ::1 entry from /etc/hosts.

I added the localhost entry back to /etc/hosts:

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

I removed the statements disabling IPv6 from /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

I rebooted for good measure, but even after reboot ipa-server-upgrade produced the same error. Indeed, IPv6 is not enabled:

$ ping6 ::1
connect: No route to host
$ ping6 localhost
connect: No route to host
$ sysctl net.ipv6.conf.all.disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 1

That makes sense. Merely removing the lines setting IPv6 to disabled didn’t actually do anything to re-enable it.

$ sudo sysctl net.ipv6.conf.all.disable_ipv6=0
net.ipv6.conf.all.disable_ipv6 = 0
$ sudo sysctl net.ipv6.conf.lo.disable_ipv6=0
net.ipv6.conf.lo.disable_ipv6 = 0

After that change, ping6 ::1 and ping6 localhost worked as expected. I left IPv6 disabled on the default interface, but noticed in ifconfig that eth0 had picked up an IPv6 address, so I disabled that:

$ sudo sysctl net.ipv6.conf.eth0.disable_ipv6=1

I also added that same line to /etc/sysctl.conf.

I ran the upgrade again:

$ sudo ipa-server-upgrade
Upgrading IPA:. Estimated time: 1 minute 30 seconds
The IPA services were upgraded
The ipa-server-upgrade command was successful

And started FreeIPA:

$ sudo ipactl start
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting httpd Service
Starting ipa-custodia Service
Starting ntpd Service
Starting pki-tomcatd Service
Starting ipa-otpd Service
ipa: INFO: The ipactl command was successful

Success! And apparently disabling IPv6 is not the best idea.

FreeIPA connection check passes, but then fails during install

One of my FreeIPA servers is on a VM that’s too small and I’ve been having problems with it. I should have known that anything that runs Java and Tomcat should have double the processing power, double the memory, and double the drive space of whatever I think it should have. Rather than merely adjust the VM settings though, I thought I would spin up a new VM with better specs and create a new replica. Should be easy, right?

I created a new CentOS 7 VM, trinculo.osric.net, and installed ipa-server 4.5.0:

$ sudo yum install ipa-server

I checked the connection from the replica target to the master:

$ sudo ipa-replica-conncheck --master=ariel.osric.net

Likewise I checked the connection from the master to the replica target:

$ sudo ipa-replica-conncheck --replica=trinculo.osric.net

Everything was successful, so on the existing master I created the replica file:

$ sudo ipa-replica-prepare --ip-address= trinculo.osric.net

I copied that over to the replica target, but the replica installer indicated a failed connection check:

$ sudo ipa-replica-install /root/replica-info-trinculo.osric.net.gpg --ip-address=
ipa.ipapython.install.cli.install_tool(CompatServerReplicaInstall): ERROR    Connection check failed!
See /var/log/ipareplica-conncheck.log for more information.
If the check results are not valid it can be skipped with --skip-conncheck parameter.

A failed connection check when the connection checks passed? Continue reading FreeIPA connection check passes, but then fails during install

FreeIPA: Failed to start pki-tomcatd Service

After a recent CentOS update, FreeIPA 4.5 failed to start with the following error message:
Failed to start pki-tomcatd Service

What changed? The following were the 3 packages updated:

  • httpd.x86_64
  • httpd-tools.x86_64
  • mod_session.x86_64

I successfully restarted FreeIPA without the pki-tomcatd service:
$ sudo ipactl start --ignore-service-failure

But it’s not ideal to run it without the PKI service. What is going on? According to the log at /var/log/pki/pki-tomcat/ca/debug:

java.lang.Exception: Certificate auditSigningCert cert-pki-ca is invalid: Invalid certificate: (-8101) Certificate type not approved for application.

Which cert is that? Where is it? How did it get created? Didn’t FreeIPA create it? Why isn’t it valid? Why doesn’t it give me any additional info?

Eventually I found the certificate location (although I don’t recall how, likely a post on the FreeIPA mailing list):
/var/lib/pki/pki-tomcat/alias -> /etc/pki/pki-tomcat/alias

I ran certutil to find out more about the certificate:
$ certutil -L -d /etc/pki/pki-tomcat/alias
certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.

That uninformative and misleading error message looked familiar to me. Indeed, I wrote a post about it 7 months ago:
certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format

$ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'auditSigningCert cert-pki-ca'

The expiration date looked fine, which was the first thing I suspected.

I did note the following, which looked interesting:
Mozilla-CA-Policy: false (attribute missing)

But after reading about that at http://mozilla.github.io/ca-policy/ it looked like it shouldn’t be needed.

Fortunately, I have another working FreeIPA replica that I had not yet upgraded, so I compared the certificates on both systems:

On the IPA replica with errors:

$ sudo certutil -L -d /etc/pki/pki-tomcat/alias

Certificate Nickname                                         Trust Attributes

caSigningCert cert-pki-ca                                    CTu,Cu,Cu
auditSigningCert cert-pki-ca                                 u,u,u
ocspSigningCert cert-pki-ca                                  u,u,u
Server-Cert cert-pki-ca                                      u,u,u
subsystemCert cert-pki-ca                                    u,u,u

On the working IPA replica:

$ sudo certutil -L -d /etc/pki/pki-tomcat/alias

Certificate Nickname                                         Trust Attributes

caSigningCert cert-pki-ca                                    CTu,Cu,Cu
Server-Cert cert-pki-ca                                      u,u,u
auditSigningCert cert-pki-ca                                 u,u,Pu
ocspSigningCert cert-pki-ca                                  u,u,u
subsystemCert cert-pki-ca                                    u,u,u

Note the P trust attribute in the latter. What does it mean? From man certutil:

-t trustargs
           Specify the trust attributes to modify in an existing certificate
           or to apply to a certificate when creating it or adding it to a
           database. There are three available trust categories for each
           certificate, expressed in the order SSL, email, object signing for
           each trust setting. In each category position, use none, any, or
           all of the attribute codes:

           ·   p - Valid peer

           ·   P - Trusted peer (implies p)

           ·   c - Valid CA

           ·   C - Trusted CA (implies c)

           ·   T - trusted CA for client authentication (ssl server only)

I modified the trust attributes of the certificate accordingly:

$ sudo certutil -M -t ',,P' -d /etc/pki/pki-tomcat/alias -n 'auditSigningCert cert-pki-ca'

I tried restarting FreeIPA again:

$ sudo ipactl restart
Stopping pki-tomcatd Service
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting ntpd Service
Restarting pki-tomcatd Service
Restarting ipa-otpd Service
ipa: INFO: The ipactl command was successful

It worked!

But why? What does the trust attribute for JAR/XPI mean? I don’t really know — I suppose it means that that the Java code we’re running should trust the certificate. Since I didn’t have this problem when I upgraded the working replica, I’m guessing that I must have done something to change it (and break it) along the way. It likely had nothing to do with the CentOS updates I applied, but I just happened to run into the problem after restarting FreeIPA post-updates.

FreeIPA 4.5.0 upgrade fails

I recently ran the usual sudo yum update, which included an upgrade of FreeIPA from version 4.4 to version 4.5. However, the upgrade reported that it failed, so I tried to run it manually afterwards:

$ sudo ipa-server-upgrade

The command above timed out, so I ran it again in verbose mode:

$ sudo ipa-server-upgrade --verbose

With the extra output, I saw it was getting stuck on the following before timing out:

ipa: DEBUG: wait_for_open_ports: localhost [8080, 8443] timeout 300

Neither 8080 nor 8443 are on the list of FreeIPA protocols/ports:

TCP 389, 636: LDAP/LDAPS
TCP 88, 464: kerberos
UDP 88, 464: kerberos
UDP 123: NTP

So what was going on?

I checked to see if those ports were listening:

$ sudo lsof -i TCP -P
TCP *:8080 (LISTEN)
TCP *:8443 (LISTEN)

$ sudo lsof -i TCP
TCP *:webcache (LISTEN)
TCP *:pcsync-https (LISTEN)

The ports were listening, so why the timeout? Fortunately I found someone else describing a similar problem on the FreeIPA mailing list, with this reply from Alexander Bokovoy, one of the FreeIPA developers:

I’m a bit tired to repeat this multiple times but FreeIPA does require IPv6 stack to be enabled in the kernel. We absolutely do. If you don’t use IPv6 stack, disable it on specific interfaces. However, there is a practical problem with the way how glibc DNS resolver works: in default configuration it always prefers IPv6 answers to IPv4 because this is actually a policy of RFC3484. As result, if you have ::1 in /etc/hosts, it will be returned first. If you don’t have ::1 on any of your interfaces (‘lo’ is a typical one), then apps cannot contact ::1 (localhost) even if those apps that use IPv6 bind to all interfaces.

FreeIPA uses modern APIs provided by glibc to listen on both IPv6 and IPv4. It simply means that FreeIPA servers bind to IPv6 addresses (on all interfaces or on a specific one, if needed) and treat IPv4 as mapped ones because IPv6 and IPv4 share the same port space on the same machine. This works transparently thanks to glibc and is a recommended way to write networking applications. See man ipv6(7) for details.


Following that suggestion, I deleted the following line from /etc/hosts:

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Sure enough, that solved the problem! This seems like a good idea for anyone who has disabled IPv6, even if you’re not running FreeIPA. Why include an entry in your hosts file that you know doesn’t work?

I had another FreeIPA server, this one running on an underpowered VM (1GB RAM, 1 CPU). Even after removing the IPv6 entry from /etc/hosts, the FreeIPA upgrade seemed to fail. The certmonger/dogtag processes would consume all of the system resources and it would freeze. All memory was in use and nearly all of the swap space as well. The CPU was running at 5000%. I gave up on it and ignored it for a while, and it turned out that letting it sit and process for a while helped. It seem to have worked itself out: FreeIPA is at the latest version and system resource utilization is very low.

FreeIPA: updating client hostname

I recently updated some CentOS 7 hostnames to better reflect their status as cattle, not pets. Part of renaming the hosts meant updating the hosts in my FreeIPA environment. RedHat’s Identity Management Guide to Renaming Machines confirms there’s no easy way to update a hostname. You need to un-enroll the and re-enroll the client.

# ipa-client-install --uninstall

# ipa-client-install --domain=osric.net --server=freeipa.osric.net --realm=FREEIPA.OSRIC.NET --principal=admin --password=T0ps3CR3T --mkhomedir -U --hostname=www-dev-01.osric.net

Kerberos authentication failed: kinit: Cannot read password while getting initial credentials

I searched for the error and found a blog post suggesting that the password had expired. Sure enough, when I checked the FreeIPA web interface, it showed that the password for the admin user had expired. I reset it via the web interface.

I tried again, using the new password:
# ipa-client-install --domain=osric.net --server=freeipa.osric.net --realm=FREEIPA.OSRIC.NET --principal=admin --password=M0r3s3CR3Ts! --mkhomedir -U --hostname=www-dev-01.osric.net

It failed with the same error message!

When I checked /var/log/ipaclient-install.log it indicated that the password was still expired. Resetting the password via the web interface forces the user to set a new password at the next login — the password expires immediately!

I ran kinit admin on the command line and used the temporary password to log in and set a new password. Then the command to re-enroll the server worked without any errors.

FreeIPA: Could not chdir to home directory /home/bbilliards: no such file or directory

I recently installed a FreeIPA server and a FreeIPA client. I generated a Kerberos ticket for a test user, Bob Billiards, on the IPA server:

# kinit bbilliards
Password for bbilliards@IPA.OSRIC.NET:

Then I attempted to ssh into the IPA client as that user. The connection was successful, but it could not find the user’s home directory:

# ssh bbilliards@ariel.osric.net
bbilliards@ariel.osric.net's password:
Could not chdir to home directory /home/bbilliards: no such file or directory

The location of the home directory was set when I created the user, as can be seen here:

# ipa user-find bbilliards
1 user matched
  User login: bbilliards
  First name: Bob
  Last name: Billiards
  Home directory: /home/bbilliards
  Login shell: /bin/sh
  Principal name: bbilliards@IPA.OSRIC.NET
  Principal alias: bbilliards@IPA.OSRIC.NET
  Email address: bbilliards@ipa.osric.net
  UID: 1110200001
  GID: 1110200001
  SSH public key fingerprint: [redacted]
  Account disabled: False
Number of entries returned 1

Shouldn’t the system be able to create the home directory automatically? It turns out it can, if you specify the --mkhomedir switch when installing the IPA client:

# ipa-client-install --mkhomedir

Now when I ssh into the machine it creates a home directory:

# ssh bbilliards@ariel.osric.net
Creating home directory for bbilliards
-sh-4.2$ pwd

You may prefer to mount a Network File System (NFS) directory as a home directory instead so that users have the same home directories across machines.

Error: Cannot contact any KDC for realm while getting initial credentials

I’ve been testing FreeIPA on a small network of CentOS 7 hosts (all virtual machines running in VirtualBox on a host-only network). After installing the IPA server on one host and creating the realm (IPA.OSRIC.NET), I installed the IPA client on one of the other hosts and tried running kinit:

# kinit admin
kinit: Cannot contact any KDC for realm 'IPA.OSRIC.NET' while getting initial credentials

Searching for that error brought me to Kinit won’t connect to a domain server. Although that did not describe the same issue, it did point me to the /etc/krb5.conf file. The realms section looked like it was missing something:

    pkinit_anchors = FILE:/etc/ipa/ca.crt


I added a kdc attribute:

    kdc = prospero.osric.net:88
    pkinit_anchors = FILE:/etc/ipa/ca.crt

No restart of any service was necessary. I ran kinit again and it worked:

# kinit admin
Password for admin@IPA.OSRIC.NET:

According to the krb5.conf documentation on realms:

The name or address of a host running a KDC for that realm. An optional port number, separated from the hostname by a colon, may be included.

I’m a Kerberos novice, but that seems like a necessary property. I’m not sure why the IPA client setup did not include it. I have a few more virtual machines to install the client on, so I’ll soon find if that behavior is consistent on subsequent installations.