FreeIPA connection check passes, but then fails during install

One of my FreeIPA servers is on a VM that’s too small and I’ve been having problems with it. I should have known that anything that runs Java and Tomcat should have double the processing power, double the memory, and double the drive space of whatever I think it should have. Rather than merely adjust the VM settings though, I thought I would spin up a new VM with better specs and create a new replica. Should be easy, right?

I created a new CentOS 7 VM, trinculo.osric.net, and installed ipa-server 4.5.0:

$ sudo yum install ipa-server

I checked the connection from the replica target to the master:

$ sudo ipa-replica-conncheck --master=ariel.osric.net

Likewise I checked the connection from the master to the replica target:

$ sudo ipa-replica-conncheck --replica=trinculo.osric.net

Everything was successful, so on the existing master I created the replica file:

$ sudo ipa-replica-prepare --ip-address=192.168.0.101 trinculo.osric.net

I copied that over to the replica target, but the replica installer indicated a failed connection check:

$ sudo ipa-replica-install /root/replica-info-trinculo.osric.net.gpg --ip-address=192.168.0.101
...
ipa.ipapython.install.cli.install_tool(CompatServerReplicaInstall): ERROR    Connection check failed!
See /var/log/ipareplica-conncheck.log for more information.
If the check results are not valid it can be skipped with --skip-conncheck parameter.

A failed connection check when the connection checks passed?

I looked at the log:

...
2017-10-20T15:55:04Z DEBUG args=/usr/sbin/ipa-replica-conncheck --master ariel.osric.net --auto-master-check --realm IPA.OSRIC.NET --hostname trinculo.osric.net --principal admin --ca-cert-file /tmp/tmp52pKZbria/real_info/ca.crt
2017-10-20T15:55:04Z DEBUG Process finished, return code=1
...

OK, the return code=1 means there was an error. The arguments to the ipa-replica-conncheck command listed contain items in addition to what I actually specified as well, although those are likely defaults and/or pulled from the replica file generated on the FreeIPA master. There was additional info in the /var/log/ipareplica-conncheck.log file though:

Connection from replica to master is OK.
Start listening on required ports for remote master check
Get credentials to log in to remote master
Check RPC connection to remote master
trying https://ariel.osric.net/ipa/json
Retrying using SSH...
Check SSH connection to remote master
ERROR: Could not SSH to remote host.
See /var/log/ipareplica-conncheck.log for more information.

2017-10-20T15:55:27Z DEBUG Starting external process
2017-10-20T15:55:27Z DEBUG args=/usr/sbin/ipa-client-install --unattended --uninstall
2017-10-20T15:55:27Z DEBUG Process finished, return code=2
2017-10-20T15:55:27Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 333, in run
    cfgr.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 366, in run
    self.validate()

  ...

2017-10-20T15:55:28Z DEBUG The ipa-replica-install command failed, exception: ScriptError: Connection check failed!

One log file, but conflicting messages!

1. Connection from replica to master is OK.
2. ERROR: Could not SSH to remote host.
3. ScriptError: Connection check failed!

Two things I was thinking:

1. If SSH is a required connection, why isn’t it part of the ipa-replica-conncheck command?
2. Maybe I should try the --skip-conncheck flag.

I had to uninstall ipa-server on the replica target first, since some portion was installed before it encountered an error:

$ sudo ipa-server-install --uninstall

Likewise, I had to remove the replica from FreeIPA on the master:

$ sudo ipa host-del trinculo.osric.net

I’d like to say that, at this point, I ran the following and everything went perfectly:

$ sudo ipa-replica-install /root/replica-info-trinculo.osric.net.gpg --ip-address=192.168.0.101 --skip-conncheck

That would be a lie, of course.

I ran into numerous additional error messages and spent a lot of time following bad leads. Eventually, while researching one of the new error messages, I ran across a thread on the FreeIPA mailing list (Issue while setting up Replication) where the user was finally successful after spinning up a new virtual machine and starting fresh! Out of frustration, I tried the same, and this time the same command worked (with --skip-conncheck):

$ sudo ipa-replica-install /root/replica-info-trinculo.osric.net.gpg --ip-address=192.168.0.101 --skip-conncheck

That was not the most satisfying solution. On the other hand, it was finally working.

Other people have warned me that a failed FreeIPA installation can leave behind various files and artifacts that cause problems when reinstalling, even after you run ipa-server-install --uninstall. I figured that was largely resolved in current versions (I’m using 4.5.0), but there may still be some issues.