FreeIPA connection check passes, but then fails during install

One of my FreeIPA servers is on a VM that’s too small and I’ve been having problems with it. I should have known that anything that runs Java and Tomcat should have double the processing power, double the memory, and double the drive space of whatever I think it should have. Rather than merely adjust the VM settings though, I thought I would spin up a new VM with better specs and create a new replica. Should be easy, right?

I created a new CentOS 7 VM,, and installed ipa-server 4.5.0:

$ sudo yum install ipa-server

I checked the connection from the replica target to the master:

$ sudo ipa-replica-conncheck

Likewise I checked the connection from the master to the replica target:

$ sudo ipa-replica-conncheck

Everything was successful, so on the existing master I created the replica file:

$ sudo ipa-replica-prepare --ip-address=

I copied that over to the replica target, but the replica installer indicated a failed connection check:

$ sudo ipa-replica-install /root/ --ip-address=
ipa.ipapython.install.cli.install_tool(CompatServerReplicaInstall): ERROR    Connection check failed!
See /var/log/ipareplica-conncheck.log for more information.
If the check results are not valid it can be skipped with --skip-conncheck parameter.

A failed connection check when the connection checks passed? Continue reading FreeIPA connection check passes, but then fails during install

Reset the iDRAC administrator password via ipmitool

In the previous post, I configured the iDRAC interface on a Dell server using ipmitool on CentOS. However, I ran into a problem, which I blame on poor user interface design:

When you log into the iDRAC web interface as root/calvin, it warns you that you are using the default username/password and prompts you to change the password. I did so by generating a random password in my password manager and pasting it into the password field.

The problem? The password can contain at most 20 characters, a limitation that is not obvious from the web interface. The password field on the iDRAC web interface truncates the password at 20 characters, and so I submitted a partial password. Then later, when I attempting to log it using the password saved in my password manager, it didn’t match. (For reasons that aren’t clear to me, submitting just the first 20 characters of the password saved in the password manager did not work either.)

I figured I was stuck and would have to go to the data center, reboot the server, and boot into the Lifecycle Controller in order to reset the iDRAC password. But I thought I’d see what I could do via ipmitool first.

From Configuring DRAC with ipmitool and ipmitool Cheatsheet:

Reset BMC/DRAC to default:

$ sudo ipmitool mc reset cold

The command was successful, but that did not reset the password for me.

From Resetting the BMC:

…you can reset the BMC to factory defaults with IPMICFG or ipmitool. Be aware that this will wipe any existing settings on the BMC that you may have set from the web interface, but excludes network settings.

# ipmitool raw 0x3c 0x40

But that did not work for me, and produced an error code. I spent some time trying to determine what the various raw hex values for ipmi meant, but that was not productive.

Eventually though I did hit upon an ipmitool command that worked:

$ sudo ipmitool user list 1
ID  Name	     Callin  Link Auth	IPMI Msg   Channel Priv Limit
1                    true    false      false      NO ACCESS
2   superuser        true    true       true       ADMINISTRATOR
3                    true    false      false      NO ACCESS

The username I configured corresponds with ID 2, so then I used ipmitool to set the password for that user:

$ sudo ipmitool user set password 2

I was prompted to enter the password, which I was then able to use to log in to the iDRAC web interface.

Using ipmitool to configure Dell iDRAC

I have a number of Dell servers in a remote data center, so I wanted to configure the iDRAC interface in order to power on the systems remotely, get troubleshooting info for Dell, etc., without going to the data center myself. I’ve never configured iDRAC except through the Lifecycle Controller via a crash-cart on bootup. I thought that I would be spending all day in the data center getting everything configured, but when I mentioned this to another sysadmin he said, “Just use ipmitool.”

I had no idea such a tool existed!

First, I installed ipmitool (I’m using CentOS):

sudo yum install ipmitool

I found a helpful website: ipmitool Cheatsheet and Configuring DRAC from ipmitool

I was a little skeptical, but I read through (most) of the ipmitool man page to make sure I had a reasonable idea what the commands would do, and then I tried one. And immediately received an error message:

$ ipmitool lan set 1 ipsrc static
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory

I checked and found that the path listed does exist:

$ ls /dev/ipmi*

Then it hit me: I need to be superuser, don’t I? That worked!

sudo ipmitool lan set 1 ipsrc static
sudo ipmitool lan set 1 ipaddr
sudo ipmitool lan set 1 netmask
sudo ipmitool lan set 1 defgw ipaddr

I was then able to connect to the IP address in a browser (it warned me there was an untrusted certificate, and I added it as a permanent exception in the browser.)

The default username/password was root/calvin. I changed both the username and password right away. Even though I have the iDRAC interfaces on an RFC 1918 subnet and behind a firewall, why take the risk of keeping the default values?

As I discovered though, pay attention to the iDRAC password restrictions. Otherwise you may need to use ipmitool to reset the iDRAC admin password.

Ansible conditional check failed

I wanted to add a check to one of my Ansible roles so that an application source would be copied and the source recompiled only if no current version existed or if the existing version did not match the expected version:

- name: Check to see if app is installed and the expected version
  command: /usr/local/app/bin/app --version
  register: version_check
  ignore_errors: True
  changed_when: "version_check.rc != 0 or {{ target_version }} not in version_check.stdout"

- name: include app install
  include: tasks/install.yml
  when: "version_check.rc != 0 or {{ target_version }} not in version_check.stdout"

I defined the target version in my role’s defaults/main.yml:

target_version: "2.5.2"

The first time I ran it, I encountered an error:

fatal: []: FAILED! => {"failed": true, "msg": "The conditional check 'version_check.rc != 0 or {{ target_version }} not in version_check.stdout' failed. The error was: error while evaluating conditional (version_check.rc != 0 or {{ target_version }} not in version_check.stdout): Unable to look up a name or access an attribute in template string ({% if version_check.rc != 0 or 2.5.2 not in version_check.stdout %} True {% else %} False {% endif %}).\nMake sure your variable name does not contain invalid characters like '-': coercing to Unicode: need string or buffer, StrictUndefined found"}

It’s a little unclear what is wrong, so I figured it was likely an issue with quotes or a lack of parentheses.

First I tried parentheses:

changed_when: "version_check.rc != 0 or ({{ target_version }} not in version_check.stdout)"

No luck.

changed_when: "version_check.rc != 0 or !('{{ target_version }}' in version_check.stdout)"

You know, trying to google and or not or or or is is tricky. Even if you add terms like Boolean logic or propositional calculus.

I tried to break it down into smaller parts:

changed_when: "version_check.rc != 0"

That worked.

changed_when: "!('{{ target_version }}' in version_check.stdout)"

A different error appeared:

template error while templating string: unexpected char u'!'

OK, that’s getting somewhere! Try a variation:

changed_when: "'{{ target_version }}' not in version_check.stdout"

It worked! But with a warning:

[WARNING]: when statements should not include jinja2 templating delimiters
such as {{ }} or {% %}. Found: ('{{ target_version }}' not in version_check.stdout)

Next try:

changed_when: "target_version not in version_check.stdout"

That worked, and without any warnings. I put the or back in:

changed_when: "version_check.rc != 0 or target_version not in version_check.stdout"

That worked! It was the jinja2 delimiters the whole time. The value of the changed_when key is already interpreted as jinja2 apparently, so the delimiters were redundant. Even though it succeeded (with a warning) in a single propositional statement, it failed when the logical disjunction was added. It was an important reminder: error messages aren’t perfect.

FreeIPA: Failed to start pki-tomcatd Service

After a recent CentOS update, FreeIPA 4.5 failed to start with the following error message:
Failed to start pki-tomcatd Service

What changed? The following were the 3 packages updated:

  • httpd.x86_64
  • httpd-tools.x86_64
  • mod_session.x86_64

I successfully restarted FreeIPA without the pki-tomcatd service:
$ sudo ipactl start --ignore-service-failure

But it’s not ideal to run it without the PKI service. What is going on? According to the log at /var/log/pki/pki-tomcat/ca/debug:

java.lang.Exception: Certificate auditSigningCert cert-pki-ca is invalid: Invalid certificate: (-8101) Certificate type not approved for application.

Which cert is that? Where is it? How did it get created? Didn’t FreeIPA create it? Why isn’t it valid? Why doesn’t it give me any additional info?

Eventually I found the certificate location (although I don’t recall how, likely a post on the FreeIPA mailing list):
/var/lib/pki/pki-tomcat/alias -> /etc/pki/pki-tomcat/alias

I ran certutil to find out more about the certificate:
$ certutil -L -d /etc/pki/pki-tomcat/alias
certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.

That uninformative and misleading error message looked familiar to me. Indeed, I wrote a post about it 7 months ago:
certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format

$ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'auditSigningCert cert-pki-ca'

The expiration date looked fine, which was the first thing I suspected.

I did note the following, which looked interesting:
Mozilla-CA-Policy: false (attribute missing)

But after reading about that at it looked like it shouldn’t be needed.

Fortunately, I have another working FreeIPA replica that I had not yet upgraded, so I compared the certificates on both systems:

On the IPA replica with errors:

$ sudo certutil -L -d /etc/pki/pki-tomcat/alias

Certificate Nickname                                         Trust Attributes

caSigningCert cert-pki-ca                                    CTu,Cu,Cu
auditSigningCert cert-pki-ca                                 u,u,u
ocspSigningCert cert-pki-ca                                  u,u,u
Server-Cert cert-pki-ca                                      u,u,u
subsystemCert cert-pki-ca                                    u,u,u

On the working IPA replica:

$ sudo certutil -L -d /etc/pki/pki-tomcat/alias

Certificate Nickname                                         Trust Attributes

caSigningCert cert-pki-ca                                    CTu,Cu,Cu
Server-Cert cert-pki-ca                                      u,u,u
auditSigningCert cert-pki-ca                                 u,u,Pu
ocspSigningCert cert-pki-ca                                  u,u,u
subsystemCert cert-pki-ca                                    u,u,u

Note the P trust attribute in the latter. What does it mean? From man certutil:

-t trustargs
           Specify the trust attributes to modify in an existing certificate
           or to apply to a certificate when creating it or adding it to a
           database. There are three available trust categories for each
           certificate, expressed in the order SSL, email, object signing for
           each trust setting. In each category position, use none, any, or
           all of the attribute codes:

           ·   p - Valid peer

           ·   P - Trusted peer (implies p)

           ·   c - Valid CA

           ·   C - Trusted CA (implies c)

           ·   T - trusted CA for client authentication (ssl server only)

I modified the trust attributes of the certificate accordingly:

$ sudo certutil -M -t ',,P' -d /etc/pki/pki-tomcat/alias -n 'auditSigningCert cert-pki-ca'

I tried restarting FreeIPA again:

$ sudo ipactl restart
Stopping pki-tomcatd Service
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting ntpd Service
Restarting pki-tomcatd Service
Restarting ipa-otpd Service
ipa: INFO: The ipactl command was successful

It worked!

But why? What does the trust attribute for JAR/XPI mean? I don’t really know — I suppose it means that that the Java code we’re running should trust the certificate. Since I didn’t have this problem when I upgraded the working replica, I’m guessing that I must have done something to change it (and break it) along the way. It likely had nothing to do with the CentOS updates I applied, but I just happened to run into the problem after restarting FreeIPA post-updates.

Guest SSID surprises on home wireless router

My current home Internet provider is CenturyLink, and with that I’m using their recommended Zyxel C1100Z “modem”.

Via the modem’s web interface you can configure up to 4 SSIDs. I have one set up for my devices with strong security settings, and another set up for guests with weaker security settings. One thing that surprised me: when I checked the list of attached devices, devices attached to the guest SSID were allocated IP addresses in the same address range as, and could communicate with, devices attached to my trusted home SSID.

The Zyxel C1100Z will let you create LAN subnets with different IP address ranges and settings, but a device on one subnet can still communicate with devices on another LAN subnet. This would let you at least configure a host firewall (on hosts that support a host firewall) to drop traffic from a particular address range (e.g.

This is lunacy, though. Why would you create separate SSIDs with different security settings if the attached devices cannot be isolated from one another? I suspect that most users do not realize this. There are some settings you can change from one SSID to another, such as bandwidth throttling, but that seems like a secondary consideration to securing your network. Needless to say, my guest network has the same security settings as my trusted home network now.

I wondered if I had overlooked a setting somewhere, so I called to confirm with CenturyLink. The technician there was able to identify the SSIDs I had configured, suggesting that they have a backdoor into the modem they provided.

The moral of the story is: never use the equipment provided by your ISP.

FreeIPA 4.5.0 upgrade fails

I recently ran the usual sudo yum update, which included an upgrade of FreeIPA from version 4.4 to version 4.5. However, the upgrade reported that it failed, so I tried to run it manually afterwards:

$ sudo ipa-server-upgrade

The command above timed out, so I ran it again in verbose mode:

$ sudo ipa-server-upgrade --verbose

With the extra output, I saw it was getting stuck on the following before timing out:

ipa: DEBUG: wait_for_open_ports: localhost [8080, 8443] timeout 300

Neither 8080 nor 8443 are on the list of FreeIPA protocols/ports:

TCP 389, 636: LDAP/LDAPS
TCP 88, 464: kerberos
UDP 88, 464: kerberos
UDP 123: NTP

So what was going on?

I checked to see if those ports were listening:

$ sudo lsof -i TCP -P
TCP *:8080 (LISTEN)
TCP *:8443 (LISTEN)

$ sudo lsof -i TCP
TCP *:webcache (LISTEN)
TCP *:pcsync-https (LISTEN)

The ports were listening, so why the timeout? Fortunately I found someone else describing a similar problem on the FreeIPA mailing list, with this reply from Alexander Bokovoy, one of the FreeIPA developers:

I’m a bit tired to repeat this multiple times but FreeIPA does require IPv6 stack to be enabled in the kernel. We absolutely do. If you don’t use IPv6 stack, disable it on specific interfaces. However, there is a practical problem with the way how glibc DNS resolver works: in default configuration it always prefers IPv6 answers to IPv4 because this is actually a policy of RFC3484. As result, if you have ::1 in /etc/hosts, it will be returned first. If you don’t have ::1 on any of your interfaces (‘lo’ is a typical one), then apps cannot contact ::1 (localhost) even if those apps that use IPv6 bind to all interfaces.

FreeIPA uses modern APIs provided by glibc to listen on both IPv6 and IPv4. It simply means that FreeIPA servers bind to IPv6 addresses (on all interfaces or on a specific one, if needed) and treat IPv4 as mapped ones because IPv6 and IPv4 share the same port space on the same machine. This works transparently thanks to glibc and is a recommended way to write networking applications. See man ipv6(7) for details.


Following that suggestion, I deleted the following line from /etc/hosts:

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Sure enough, that solved the problem! This seems like a good idea for anyone who has disabled IPv6, even if you’re not running FreeIPA. Why include an entry in your hosts file that you know doesn’t work?

I had another FreeIPA server, this one running on an underpowered VM (1GB RAM, 1 CPU). Even after removing the IPv6 entry from /etc/hosts, the FreeIPA upgrade seemed to fail. The certmonger/dogtag processes would consume all of the system resources and it would freeze. All memory was in use and nearly all of the swap space as well. The CPU was running at 5000%. I gave up on it and ignored it for a while, and it turned out that letting it sit and process for a while helped. It seem to have worked itself out: FreeIPA is at the latest version and system resource utilization is very low.

Using with fail2ban

Anyone who runs a server with open ports knows that systems with questionable intent will come knocking. Sometimes thousands of them. fail2ban is software that that checks your server logs and detects multiple failures, for example 5 failed SSH logins in a row, and bans the source IP address a period of time, e.g. for an hour. This helps prevent password-guessing and brute force attacks. It might be useful to share information about those questionable IP addresses with others so that we can block them proactively.

One such list of IP addresses that I found is Since I am primarily concerned with systems that are trying to SSH into my system, I looked specifically at their SSH blocklist:
All IP addresses which have been reported within the last 48 hours as having run attacks on the service SSH.

Implementation details: Continue reading Using with fail2ban

Wifi on Raspberry Pi 3

I don’t run a GUI/Desktop on my Raspberry Pi devices. I don’t have monitors or keyboards connected to them — typically I log into them via SSH and manage them that way. I recently wanted to activate multiple network connections on my Raspberry Pi 3 Model B, so I decided to activate the wireless connection in addition to the wired connection that was already configured.

I tried following the steps at Setting WiFi up via the command line but got an error after the first step:

$ sudo iwlist wlan0 scan
wlan0     Interface doesn't support scanning : Network is down

I decided to check an see what devices were available via ifconfig:

$ ifconfig
eth0 ...
lo ...

No wireless device is listed there. I checked for all devices using the -a switch:

$ ifconfig -a
eth0 ...
lo ...
wlan0 ...

It’s there, wlan0, but it wasn’t active. I tried to bring the device up:

$ sudo ifconfig wlan0 up
SIOCSIFFLAGS: Operation not possible due to RF-kill

That error message is completely indecipherable to me! Fortunately, someone else had this error message too:
“SIOCSIFFLAGS: Operation not possible due to RF-kill”?

From that post I was able to determine that the wireless device was soft blocked:

$ sudo rfkill list
0: phy0: Wireless LAN
        Soft blocked: yes
        Hard blocked: no
1: hci0: Bluetooth
        Soft blocked: yes
        Hard blocked: no

How to unblock it? The following, described at the aforementioned post, worked. Although I’m not sure why I’m unblocking wifi instead of phy0:

$ sudo rfkill unblock wifi
$ sudo rfkill list
0: phy0: Wireless LAN
        Soft blocked: no
        Hard blocked: no
1: hci0: Bluetooth
        Soft blocked: yes
        Hard blocked: no

$ sudo ifconfig wlan0 up
$ ifconfig
eth0 ...
lo ...
wlan0 ...

The device is now active, but we still need to connect. First, I generated an encrypted passphrase:

$ wpa_passphrase "my_network_id" "my_network_password"

Next, I appended that block of text to /etc/wpa_supplicant/wpa_supplicant.conf

Based on what I’ve read, that should be sufficient. The system should periodically detect changes to the wpa_supplicant.conf file and load the new settings automatically. I was impatient and rebooted. Apparently the following command should work too:

$ sudo wpa_cli reconfigure

After the reboot, I checked ifconfig for the wlan0 interface:

$ ifconfig wlan0 | grep 'inet addr'
          inet addr:  Bcast:  Mask:

It has an address — success!

Nagios check_disk returns DISK CRITICAL – /sys/kernel/config is not accessible: Permission denied

I enabled Nagios checks for free disk space on a group of servers today, and was hit with alerts containing the following error message:
DISK CRITICAL - /sys/kernel/config is not accessible: Permission denied

If you are looking for a solution, skip to the end. Some of my mistakes before finding the solution may be interesting though!

Continue reading Nagios check_disk returns DISK CRITICAL – /sys/kernel/config is not accessible: Permission denied