Let’s Encrypt: certbot error “No vhost exists with servername or alias of”

It’s about time–or rather, years past time–I enabled HTTPS for this site. I decided to try Let’s Encrypt. It wasn’t as turnkey as I expected, so I’ve included some notes here in case anyone else has similar issues.

The Let’s Encrypt site suggested installing Certbot and included specific instructions for using Certbot with Apache on CentOS 7. It suggested that a single command might do the trick:

$ sudo certbot --apache

Unfortunately, I received a couple error messages and it was ultimately able to create the certificate for me, but unable to update my Apache configuration. An excerpt of the output of the certbot command is below:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
No names were found in your configuration files. Please enter in your domain
name(s) (comma and/or space separated) (Enter 'c' to cancel):osric.com,www.osric.com
...
No vhost exists with servername or alias of: osric.com (or it's in a file with multiple vhosts, which Certbot can't parse yet). No vhost was selected. Please specify ServerName or ServerAlias in the Apache config, or split vhosts into separate files.
Falling back to default vhost *:443...
No vhost exists with servername or alias of: www.osric.com (or it's in a file with multiple vhosts, which Certbot can't parse yet). No vhost was selected. Please specify ServerName or ServerAlias in the Apache config, or split vhosts into separate files.
Falling back to default vhost *:443...
...
No vhost selected

IMPORTANT NOTES:
- Unable to install the certificate
...

I’m guessing it’s because my Apache virtual host configuration is in /etc/httpd/conf/vhosts/chris/osric.com instead of the expected location.

I looked at the certbot documentation hoping to find a way I could pass the certbot command the path to my virtual host configuration file. I did not find an option to do that. The logs at /var/log/letsencrypt/letsencrypt.log are fairly verbose, but it still does not indicate what files or directories it looked at to attempt to find my Apache configuration.

I noted that /etc/letsencrypt/options-ssl-apache.conf contains Apache directives. I thought maybe I could just include it in my config file using Apache’s Include directive, e.g.:

Include /etc/letsencrypt/options-ssl-apache.conf

I restarted Apache using systemctl (I know, I should be using apachectl restart instead):

$ sudo systemctl restart httpd
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.

Two problems there. One, options-ssl-apache.conf appears to be a generic file with no data specific to the host or cert. Additionally, I had just added it to a VirtualHost directive listening on port 80.

I duplicated the VirtualHost directive in my config file at /etc/httpd/conf/vhosts/chris/osric.com and made a few modifications and additions:

<IfModule mod_ssl.c>
<VirtualHost 216.154.220.53:443>
...all the directives from the port 80 VirtualHost...
SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/osric.com/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/osric.com/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/osric.com/chain.pem
</VirtualHost>
</IfModule>

I restarted Apache:

$ sudo apachectl restart

The server restarted, but still did not respond to HTTPS requests. It didn’t appear to be listening on 443:

$ curl https://www.osric.com
curl: (7) Failed connect to www.osric.com:443; Connection refused

As a sanity check, I confirmed that mod_ssl was indeed installed:

$ yum list mod_ssl
Installed Packages
mod_ssl.x86_64 1:2.4.6-45.el7.centos @base

And I checked to confirm that Apache was loading mod_ssl:

$ cat /etc/httpd/conf.modules.d/00-ssl.conf
LoadModule ssl_module modules/mod_ssl.so

I looked at some other Apache configurations where I knew SSL was working and I noted the Listen directive:

Listen 443

I added that line to the top of my configuration file at /etc/httpd/conf/vhosts/chris/osric.com, above the VirtualHost directive. I restarted Apache and it worked!

FreeIPA: updating client hostname

I recently updated some CentOS 7 hostnames to better reflect their status as cattle, not pets. Part of renaming the hosts meant updating the hosts in my FreeIPA environment. RedHat’s Identity Management Guide to Renaming Machines confirms there’s no easy way to update a hostname. You need to un-enroll the and re-enroll the client.

Un-enroll:
# ipa-client-install --uninstall

Re-enroll:
# ipa-client-install --domain=osric.net --server=freeipa.osric.net --realm=FREEIPA.OSRIC.NET --principal=admin --password=T0ps3CR3T --mkhomedir -U --hostname=www-dev-01.osric.net

Error:
Kerberos authentication failed: kinit: Cannot read password while getting initial credentials

I searched for the error and found a blog post suggesting that the password had expired. Sure enough, when I checked the FreeIPA web interface, it showed that the password for the admin user had expired. I reset it via the web interface.

I tried again, using the new password:
# ipa-client-install --domain=osric.net --server=freeipa.osric.net --realm=FREEIPA.OSRIC.NET --principal=admin --password=M0r3s3CR3Ts! --mkhomedir -U --hostname=www-dev-01.osric.net

It failed with the same error message!

When I checked /var/log/ipaclient-install.log it indicated that the password was still expired. Resetting the password via the web interface forces the user to set a new password at the next login — the password expires immediately!

I ran kinit admin on the command line and used the temporary password to log in and set a new password. Then the command to re-enroll the server worked without any errors.

certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.

I was attempting to view the certificate for my FreeIPA server:

$ certutil -L -n 'IPA CA' -d /etc/dirsrv/slapd-FREEIPA-OSRIC-NET/
certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.

That had me worried. Was my cert/key database corrupt? Turns out, I just didn’t have permission to read the files. It worked when I tried it with sudo:

$ sudo certutil -L -n 'IPA CA' -d /etc/dirsrv/slapd-FREEIPA-OSRIC-NET/

That produced the expected output.

The old, unsupported format error is produced in a variety of cases and is often not helpful or informative. Permissions are just one reason why you might run into this message. Other reasons I’ve found include specifying a directory that does not contain the expected cert database files (i.e. cert8.db, key3.db, and secmod.db), or specifying a directory that does not exist.

VirtualBox static IP address on a host-only network

I have a number of CentOS 7 servers that comprise a FreeIPA domain on a VirtualBox host-only network. Whenever I start a server though, it is liable to pick up an IP address that I’ve already assigned to another server (which is currently powered off) in /etc/hosts.

How do I assign it a specific static IP address?

In CentOS 7, you can use the Network Manager Text User Interface (nmtui) to edit the network settings. Here’s the first thing I tried, which wasn’t quite right:

# nmtui

  • Edit a connection
  • Select a connection, e.g. enp0s3
  • IPv4 Configuration
  • Change from Automatic to Manual
  • Select Show
  • Enter 192.168.56.109/32 for addresses
  • Enter 192.168.56.255 for the gateway

When I used those settings, it didn’t work. No route to host, etc. I looked at the network interface settings via a different method:

# ip addr show

The brd (broadcast) address listed was the same as my ip address, 192.168.56.109, which was unexpected and probably why it wasn’t working!

I ran nmtui again and changed the address from 192.168.56.109/32 to 192.168.56.109/24 and it worked.

Since the /32 is interpreted as the subnet mask, it created a subnet with an address range of 1, and the broadcast address would be the same as the ip address. Specifying a subnet mask of /24 creates a subnet with 256 addresses, and a broadcast address of 192.168.56.255 (the same as was listed for the other machines on the virtual network that were using DHCP).

VMWare VSphere CLI vmware-cmd and the cfg parameter

I have a VMWare ESXi host. I can manage it through VMWare Fusion, although the options seem limited (I’m used to using vCenter Server, but I don’t have the license for that in this environment). I thought I’d give the VMWare vSphere Command Line Interface (CLI) a try. This was a mistake, but if you insist on following me down the same path, see the Drivers and Tools section on the VMWare vSphere Downloads page to get started.

First I tried vmware-cmd.

C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd
'vmware-cmd' is not recognized as an internal or external command,
operable program or batch file.

The actual file is vmware-cmd.pl (it’s in the bin folder).

I was able to run one command, to list the virtual machines on the host:
C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd.pl -H esxi.osric.net -l
Enter username: chris
Enter password:

The documentation I was looking at was probably outdated, as the newer documentation gives better examples. But the version I was looking at indicated that most of the other commands require a <config_file_path> or <cfg> parameter. Unfortunately, it does not specify what those values consist of or what they might look like. There was a hint in the docs in vmware-cmd Overview:

vmware-cmd is a legacy tool and supports the usage of VMFS paths for virtual machine configuration files. As a rule, use datastore paths to access virtual machine configuration files.

It appears that <cfg> is the path to the VMX. There are several different ways to specify this:

Full path using GUID
C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd.pl -H esxi.osric.net /vmfs/volumes/272c880d-a89548c1-a530-4bccbbad9507/benvolio/benvolio.vmx uptime
Enter username: chris
Enter password:
getuptime() = 7193

(The GUID is displayed in the output of the list of virtual machines from vmware-cmd.pl -l.)

Full path using Datastore Name
C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd.pl -H esxi.osric.net "/vmfs/volumes/test vms/benvolio/benvolio.vmx" uptime
Enter username: chris
Enter password:
getuptime() = 7578

Datastore Name + relative path
C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd.pl -H esxi.osric.net "[test vms] benvolio/benvolio.vmx" uptime
Enter username: chris
Enter password:
getuptime() = 7822

Entering my username and password every time is tedious though. According to the Connection Options for vmware-cmd:

The vmware-cmd vCLI command supports only a specific set of connection options. Other vCLI connection options are not supported, for example, you cannot use variables because the corresponding option is not supported.

In this case, I have the vSphere CLI installed on a password-protected Windows 2012r2 virtual machine, so I didn’t feel it was too much of a risk to set a temporary environment variable to store some of the connection options:

C:\Program Files (x86)\VMware\VMware vSphere CLI>SET VMOPTIONS=-H esxi.osric.net -U chris -P t0u6hpa55w0rd
C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd.pl %VMOPTIONS% -l

Remember how the documentation said that “vmware-cmd is a legacy tool”?

I’m not sure what the official replacement is–possibly the PowerShell-based VMWare vSphere PowerCLI–but it turns out that the vSphere Client is free. Accessing your ESXi host via HTTPS should provide a link to download the installer. The vSphere Client does not appear to be something you can script against or automate, but for simple tasks it is much easier to use than vmware-cmd.pl.

FreeIPA: Could not chdir to home directory /home/bbilliards: no such file or directory

I recently installed a FreeIPA server and a FreeIPA client. I generated a Kerberos ticket for a test user, Bob Billiards, on the IPA server:

# kinit bbilliards
Password for bbilliards@IPA.OSRIC.NET:

Then I attempted to ssh into the IPA client as that user. The connection was successful, but it could not find the user’s home directory:

# ssh bbilliards@ariel.osric.net
bbilliards@ariel.osric.net's password:
Could not chdir to home directory /home/bbilliards: no such file or directory

The location of the home directory was set when I created the user, as can be seen here:

# ipa user-find bbilliards
--------------
1 user matched
--------------
  User login: bbilliards
  First name: Bob
  Last name: Billiards
  Home directory: /home/bbilliards
  Login shell: /bin/sh
  Principal name: bbilliards@IPA.OSRIC.NET
  Principal alias: bbilliards@IPA.OSRIC.NET
  Email address: bbilliards@ipa.osric.net
  UID: 1110200001
  GID: 1110200001
  SSH public key fingerprint: [redacted]
  Account disabled: False
----------------------------
Number of entries returned 1
----------------------------

Shouldn’t the system be able to create the home directory automatically? It turns out it can, if you specify the --mkhomedir switch when installing the IPA client:

# ipa-client-install --mkhomedir

Now when I ssh into the machine it creates a home directory:

# ssh bbilliards@ariel.osric.net
Creating home directory for bbilliards
-sh-4.2$ pwd
/home/bbilliards

You may prefer to mount a Network File System (NFS) directory as a home directory instead so that users have the same home directories across machines.

Error: Cannot contact any KDC for realm while getting initial credentials

I’ve been testing FreeIPA on a small network of CentOS 7 hosts (all virtual machines running in VirtualBox on a host-only network). After installing the IPA server on one host and creating the realm (IPA.OSRIC.NET), I installed the IPA client on one of the other hosts and tried running kinit:

# kinit admin
kinit: Cannot contact any KDC for realm 'IPA.OSRIC.NET' while getting initial credentials

Searching for that error brought me to Kinit won’t connect to a domain server. Although that did not describe the same issue, it did point me to the /etc/krb5.conf file. The realms section looked like it was missing something:

[realms]
  IPA.OSRIC.NET = {
    pkinit_anchors = FILE:/etc/ipa/ca.crt

  }

I added a kdc attribute:

[realms]
  IPA.OSRIC.NET = {
    kdc = prospero.osric.net:88
    pkinit_anchors = FILE:/etc/ipa/ca.crt
 
  }

No restart of any service was necessary. I ran kinit again and it worked:

# kinit admin
Password for admin@IPA.OSRIC.NET:

According to the krb5.conf documentation on realms:

kdc
The name or address of a host running a KDC for that realm. An optional port number, separated from the hostname by a colon, may be included.

I’m a Kerberos novice, but that seems like a necessary property. I’m not sure why the IPA client setup did not include it. I have a few more virtual machines to install the client on, so I’ll soon find if that behavior is consistent on subsequent installations.

check_http returns 403 Forbidden on fresh Nagios installation

I recently installed a Nagios server on a new CentOS 7 virtual machine (on Virtual Box).

One of the default checks included upon installation is a check on localhost to confirm that the HTTP server is responding. (First I had to install the check_http plugin, see previous post.) The Nagios web interface reports a warning for this check:

HTTP WARNING: HTTP/1.1 403 Forbidden - 5261 bytes in 0.001 second response time

This is unexpected, since I can request the same page in a browser, which returns the Apache Welcome page.

When I run the check manually I get the same result, as expected:

# /usr/lib64/nagios/plugins/check_http -H localhost
HTTP WARNING: HTTP/1.1 403 Forbidden - 5261 bytes in 0.001 second response time |time=0.000907s|;;;0.000000 size 5261B;;;0

I checked with curl:

# curl http://localhost

This returns the HTML source of the Apache Welcome page. It looks like it is working, right? But looking at the headers returned by the Apache server also shows 403 Forbidden:

# curl -I http://localhost
HTTP/1.1 403 Forbidden
...

The Apache Welcome page gives some hints about this behavior:

Are you the Administrator?

You should add your website content to the directory /var/www/html/.

To prevent this page from ever being used, follow the instructions in the file /etc/httpd/conf.d/welcome.conf.

The /etc/httpd/conf.d/welcome.conf file begins with the following comments and directive:

#
# This configuration file enables the default "Welcome" page if there
# is no default index page present for the root URL.  To disable the
# Welcome page, comment out all the lines below.
#
# NOTE: if this file is removed, it will be restored on upgrades.
#
<LocationMatch "^/+$">
    Options -Indexes
    ErrorDocument 403 /.noindex.html
</LocationMatch>

The Apache config is specifying that if there is no index page for the document root, return the Welcome page as an error document with a 403 HTTP status code.

Once I added a basic HTML file at /var/www/html/index.html, Nagios returned a success message:

HTTP OK: HTTP/1.1 200 OK - 549 bytes in 0.001 second response time

Missing Nagios plugins in CentOS 7

I set up a Nagios server on a CentOS 7 VM (Virtual Machine):

sudo yum install epel-release
sudo yum install nrpe
sudo yum install nagios

By default it sets up some basic checks for localhost. When I checked the Nagios site at http://127.0.0.1/nagios/, I found that even PING was critical:

(No output on stdout) stderr: execvp(/usr/lib64/nagios/plugins/check_ping, ...) failed. errno is 2: No such file or directory

I checked the contents of the plugins directory:

# ls /usr/lib64/nagios/plugins
eventhandlers negate urlize utils.sh

Sure enough, the usual suspects are not there. E.g.:

  • check_load
  • check_ping
  • check_disk
  • check_http
  • check_procs

Eventually I stumbled onto the following document, /usr/share/doc/nagios-plugins-2.0.3/README.Fedora:

Fedora users

Nagios plugins for Fedora have all been packaged separately. For
example, to isntall the check_http just install nagios-plugins-http.

All plugins are installed in the architecture dependent directory
/usr/lib{,64}/nagios/plugins/.

I installed some of the plugins following that convention:

sudo yum install nagios-plugins-load
sudo yum install nagios-plugins-ping
sudo yum install nagios-plugins-disk
sudo yum install nagios-plugins-http
sudo yum install nagios-plugins-procs

Now the the corresponding plugins exist in /usr/lib64/nagios/plugins, and Nagios reports OK for those checks on localhost.

Browser metadata phishing?

I was checking my Google Analytics stats and noticed a strange entry in the Languages section of the demographics. Ranking fifth, after en-us, en-gb, en-ca, and en-au was the following:

Secret.ɢoogle.com You are invited! Enter only with this ticket URL. Copy it. Vote for Trump!

Do not visit that URL, by the way. You can see that the first “G” in “Google” is an unusual character — it’s the symbol for a voiced uvular stop.

I usually use urlQuery to check out potentially malicious sites, but it didn’t like this URL. I used vURL Online instead, which reported it was malicious:

This domain is listed in the Malware Domain List. Website’s [sic] in this database should be viewed with extreme caution.

These 1500 or so sessions on my site are presumably from some hijacked browser or malicious plug-in/extension, and the end-user has no idea they are sending this bizarre language string in the HTTP headers.

Why put a malicious URL there at all? Did the creator hope that those of us perusing our web stats would be intrigued enough to fall for this trap? Even as I ask that question, I know that some percentage of users must have done just that. I assume they are now broadcasting their language as the same unusual string.

As a site owner, is there anything I should do? I could detect this string and notify the user. E.g. use an Apache re-write rule to redirect the user to a page telling them their browser is infected? This is only a partially rhetorical question. If you have suggestions, let me know.