Tag: centos 7

yum Error: requested datatype primary not available

I ran into a new-to-me yum error earlier today:

$ yum --quiet check-updates
Error: requested datatype primary not available

Following the tips on Unix & Linux StackExchange: Error: requested datatype primary not available, I:

  • ran yum clean all
  • disabled repositories one at a time to identify the repo that was causing the error

In my case, it turned out to be the extras repo. The following did not produce any errors:

$ yum --quiet --disablerepo=extras check-updates

What is wrong with the extras repo? It is defined in /etc/yum.repos.d/CentOS-Base.repo, so I took a look at what was there:

[extras]
name=CentOS-$releasever - Extras
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

None of that looked unusual (or had changed recently), so back to Google.

I tried excluding the specific mirror that was listed for the extras repo (http://mirrors.unifiedlayer.com/centos/7.4.1708/extras/x86_64/) by adding unifiedlayer.com to the exclude line in /etc/yum/pluginconf.d/fastestmirror.conf, as described in yum and fastestmirror plugin. Although yum appeared to pick a different mirror it still gave me the same error.

It turns out, the mirror in question was “poisoned” (rerouted) by my DNS servers, as it had been identified (possibly erroneously) as malicious. As such, the domain still resolved but the path to the CentOS repository did not exist.

I didn’t think that excluding the domain in fastestmirror.conf was having the intended effect, and yum was still trying to contact the bad mirror. I took the following steps, which resolved the error, although I can’t say I entirely understand why:

$ sudo yum makecache

This still produced the error.

I removed the bad entry from:

/var/cache/yum/x86_64/7/extras/mirrorlist.txt

Then I ran makecache again:

$ sudo yum makecache

No error this time! I tried running check-update:

$ yum check-update

No error!

Shouldn’t yum clean all have eliminated the bad cache value in /var/cache/yum/x86_64/7/extras/mirrorlist.txt?

Cache invalidation, one of the hard problems. At least I have steps to take if I run into this problem again.

Reset the iDRAC administrator password via ipmitool

In the previous post, I configured the iDRAC interface on a Dell server using ipmitool on CentOS. However, I ran into a problem, which I blame on poor user interface design:

When you log into the iDRAC web interface as root/calvin, it warns you that you are using the default username/password and prompts you to change the password. I did so by generating a random password in my password manager and pasting it into the password field.

The problem? The password can contain at most 20 characters, a limitation that is not obvious from the web interface. The password field on the iDRAC web interface truncates the password at 20 characters, and so I submitted a partial password. Then later, when I attempting to log it using the password saved in my password manager, it didn’t match. (For reasons that aren’t clear to me, submitting just the first 20 characters of the password saved in the password manager did not work either.)

I figured I was stuck and would have to go to the data center, reboot the server, and boot into the Lifecycle Controller in order to reset the iDRAC password. But I thought I’d see what I could do via ipmitool first.

From Configuring DRAC with ipmitool and ipmitool Cheatsheet:

Reset BMC/DRAC to default:

$ sudo ipmitool mc reset cold

The command was successful, but that did not reset the password for me.

From Resetting the BMC:

…you can reset the BMC to factory defaults with IPMICFG or ipmitool. Be aware that this will wipe any existing settings on the BMC that you may have set from the web interface, but excludes network settings.

# ipmitool raw 0x3c 0x40

But that did not work for me, and produced an error code. I spent some time trying to determine what the various raw hex values for ipmi meant, but that was not productive.

Eventually though I did hit upon an ipmitool command that worked:

$ sudo ipmitool user list 1
ID  Name	     Callin  Link Auth	IPMI Msg   Channel Priv Limit
1                    true    false      false      NO ACCESS
2   superuser        true    true       true       ADMINISTRATOR
3                    true    false      false      NO ACCESS
etc.

The username I configured corresponds with ID 2, so then I used ipmitool to set the password for that user:

$ sudo ipmitool user set password 2

I was prompted to enter the password, which I was then able to use to log in to the iDRAC web interface.

Using ipmitool to configure Dell iDRAC

I have a number of Dell servers in a remote data center, so I wanted to configure the iDRAC interface in order to power on the systems remotely, get troubleshooting info for Dell, etc., without going to the data center myself. I’ve never configured iDRAC except through the Lifecycle Controller via a crash-cart on bootup. I thought that I would be spending all day in the data center getting everything configured, but when I mentioned this to another sysadmin he said, “Just use ipmitool.”

I had no idea such a tool existed!

First, I installed ipmitool (I’m using CentOS):

sudo yum install ipmitool

I found a helpful website: ipmitool Cheatsheet and Configuring DRAC from ipmitool

I was a little skeptical, but I read through (most) of the ipmitool man page to make sure I had a reasonable idea what the commands would do, and then I tried one. And immediately received an error message:

$ ipmitool lan set 1 ipsrc static
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory

I checked and found that the path listed does exist:

$ ls /dev/ipmi*
/dev/ipmi0

Then it hit me: I need to be superuser, don’t I? That worked!

sudo ipmitool lan set 1 ipsrc static
sudo ipmitool lan set 1 ipaddr 192.168.100.1
sudo ipmitool lan set 1 netmask 255.255.255.0
sudo ipmitool lan set 1 defgw ipaddr 192.168.100.254

I was then able to connect to the IP address in a browser (it warned me there was an untrusted certificate, and I added it as a permanent exception in the browser.)

The default username/password was root/calvin. I changed both the username and password right away. Even though I have the iDRAC interfaces on an RFC 1918 subnet and behind a firewall, why take the risk of keeping the default values?

As I discovered though, pay attention to the iDRAC password restrictions. Otherwise you may need to use ipmitool to reset the iDRAC admin password.

Using fail2ban with iptables instead of firewalld

In the previous post I wrote about the minor configuration changes needed to get fail2ban to actually do something.

I have been working primarily with CentOS 7 and have been using iptables instead of firewalld. Normally, fail2ban works with iptables by default. However, installing fail2ban on CentOS 7 also installs fail2ban-firewalld — which changes that default. Even with a properly configured fail2ban jail, you will not see the expected results. fail2ban will log events as expected, but no traffic will actually be banned.

The fail2ban-firewalld package places a file in /etc/fail2ban/jail.d/00-firewalld.conf. It overrides the default banaction (iptables) and sets it to firewallcmd-ipset.

The top of the 00-firewalld.conf file says:

You can remove this package (along with the empty fail2ban meta-package) if you do not use firewalld

When I tried removing fail2ban-firewalld, it removed fail2ban as a dependency. I have a feeling the referenced fail2ban meta-package may have something to so with that.

I have not yet investigated the meta-package and de-coupling fail2ban-firewalld from fail2ban (see Update below). My solution, for now, has been to move 00-firewalld.conf and restart fail2ban:

$ sudo mv /etc/fail2ban/jail.d/00-firewalld.conf /etc/fail2ban/jail.d/00-firewalld.disabled
$ sudo systemctl restart fail2ban

The default banaction defined in jail.conf is no longer overridden and performs as expected:
banaction = iptables-multiport

Update
According to Fail2ban with FirewallD, The fail2ban package itself is a meta-package that contains several other packages, including fail2ban-firewalld and fail2ban-server. Removing the meta-package will not remove fail2ban-server.

If you’ve already moved 00-firewalld.conf to 00-firewalld.disabled, you’ll get a warning:
warning: file /etc/fail2ban/jail.d/00-firewalld.conf: remove failed: No such file or directory

You can ignore the warning, or remove 00-firewalld.disabled.

Let’s Encrypt: certbot error “No vhost exists with servername or alias of”

It’s about time–or rather, years past time–I enabled HTTPS for this site. I decided to try Let’s Encrypt. It wasn’t as turnkey as I expected, so I’ve included some notes here in case anyone else has similar issues.

The Let’s Encrypt site suggested installing Certbot and included specific instructions for using Certbot with Apache on CentOS 7. It suggested that a single command might do the trick:

$ sudo certbot --apache

Unfortunately, I received a couple error messages and it was ultimately able to create the certificate for me, but unable to update my Apache configuration. An excerpt of the output of the certbot command is below:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
No names were found in your configuration files. Please enter in your domain
name(s) (comma and/or space separated) (Enter 'c' to cancel):osric.com,www.osric.com
...
No vhost exists with servername or alias of: osric.com (or it's in a file with multiple vhosts, which Certbot can't parse yet). No vhost was selected. Please specify ServerName or ServerAlias in the Apache config, or split vhosts into separate files.
Falling back to default vhost *:443...
No vhost exists with servername or alias of: www.osric.com (or it's in a file with multiple vhosts, which Certbot can't parse yet). No vhost was selected. Please specify ServerName or ServerAlias in the Apache config, or split vhosts into separate files.
Falling back to default vhost *:443...
...
No vhost selected

IMPORTANT NOTES:
- Unable to install the certificate
...

I’m guessing it’s because my Apache virtual host configuration is in /etc/httpd/conf/vhosts/chris/osric.com instead of the expected location.

I looked at the certbot documentation hoping to find a way I could pass the certbot command the path to my virtual host configuration file. I did not find an option to do that. The logs at /var/log/letsencrypt/letsencrypt.log are fairly verbose, but it still does not indicate what files or directories it looked at to attempt to find my Apache configuration.

I noted that /etc/letsencrypt/options-ssl-apache.conf contains Apache directives. I thought maybe I could just include it in my config file using Apache’s Include directive, e.g.:

Include /etc/letsencrypt/options-ssl-apache.conf

I restarted Apache using systemctl (I know, I should be using apachectl restart instead):

$ sudo systemctl restart httpd
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.

Two problems there. One, options-ssl-apache.conf appears to be a generic file with no data specific to the host or cert. Additionally, I had just added it to a VirtualHost directive listening on port 80.

I duplicated the VirtualHost directive in my config file at /etc/httpd/conf/vhosts/chris/osric.com and made a few modifications and additions:

<IfModule mod_ssl.c>
<VirtualHost 216.154.220.53:443>
...all the directives from the port 80 VirtualHost...
SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/osric.com/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/osric.com/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/osric.com/chain.pem
</VirtualHost>
</IfModule>

I restarted Apache:

$ sudo apachectl restart

The server restarted, but still did not respond to HTTPS requests. It didn’t appear to be listening on 443:

$ curl https://www.osric.com
curl: (7) Failed connect to www.osric.com:443; Connection refused

As a sanity check, I confirmed that mod_ssl was indeed installed:

$ yum list mod_ssl
Installed Packages
mod_ssl.x86_64 1:2.4.6-45.el7.centos @base

And I checked to confirm that Apache was loading mod_ssl:

$ cat /etc/httpd/conf.modules.d/00-ssl.conf
LoadModule ssl_module modules/mod_ssl.so

I looked at some other Apache configurations where I knew SSL was working and I noted the Listen directive:

Listen 443

I added that line to the top of my configuration file at /etc/httpd/conf/vhosts/chris/osric.com, above the VirtualHost directive. I restarted Apache and it worked!

VirtualBox static IP address on a host-only network

I have a number of CentOS 7 servers that comprise a FreeIPA domain on a VirtualBox host-only network. Whenever I start a server though, it is liable to pick up an IP address that I’ve already assigned to another server (which is currently powered off) in /etc/hosts.

How do I assign it a specific static IP address?

In CentOS 7, you can use the Network Manager Text User Interface (nmtui) to edit the network settings. Here’s the first thing I tried, which wasn’t quite right:

# nmtui

  • Edit a connection
  • Select a connection, e.g. enp0s3
  • IPv4 Configuration
  • Change from Automatic to Manual
  • Select Show
  • Enter 192.168.56.109/32 for addresses
  • Enter 192.168.56.255 for the gateway

When I used those settings, it didn’t work. No route to host, etc. I looked at the network interface settings via a different method:

# ip addr show

The brd (broadcast) address listed was the same as my ip address, 192.168.56.109, which was unexpected and probably why it wasn’t working!

I ran nmtui again and changed the address from 192.168.56.109/32 to 192.168.56.109/24 and it worked.

Since the /32 is interpreted as the subnet mask, it created a subnet with an address range of 1, and the broadcast address would be the same as the ip address. Specifying a subnet mask of /24 creates a subnet with 256 addresses, and a broadcast address of 192.168.56.255 (the same as was listed for the other machines on the virtual network that were using DHCP).

Error: Cannot contact any KDC for realm while getting initial credentials

I’ve been testing FreeIPA on a small network of CentOS 7 hosts (all virtual machines running in VirtualBox on a host-only network). After installing the IPA server on one host and creating the realm (IPA.OSRIC.NET), I installed the IPA client on one of the other hosts and tried running kinit:

# kinit admin
kinit: Cannot contact any KDC for realm 'IPA.OSRIC.NET' while getting initial credentials

Searching for that error brought me to Kinit won’t connect to a domain server. Although that did not describe the same issue, it did point me to the /etc/krb5.conf file. The realms section looked like it was missing something:

[realms]
  IPA.OSRIC.NET = {
    pkinit_anchors = FILE:/etc/ipa/ca.crt

  }

I added a kdc attribute:

[realms]
  IPA.OSRIC.NET = {
    kdc = prospero.osric.net:88
    pkinit_anchors = FILE:/etc/ipa/ca.crt
 
  }

No restart of any service was necessary. I ran kinit again and it worked:

# kinit admin
Password for admin@IPA.OSRIC.NET:

According to the krb5.conf documentation on realms:

kdc
The name or address of a host running a KDC for that realm. An optional port number, separated from the hostname by a colon, may be included.

I’m a Kerberos novice, but that seems like a necessary property. I’m not sure why the IPA client setup did not include it. I have a few more virtual machines to install the client on, so I’ll soon find if that behavior is consistent on subsequent installations.

Missing Nagios plugins in CentOS 7

I set up a Nagios server on a CentOS 7 VM (Virtual Machine):

sudo yum install epel-release
sudo yum install nrpe
sudo yum install nagios

By default it sets up some basic checks for localhost. When I checked the Nagios site at http://127.0.0.1/nagios/, I found that even PING was critical:

(No output on stdout) stderr: execvp(/usr/lib64/nagios/plugins/check_ping, ...) failed. errno is 2: No such file or directory

I checked the contents of the plugins directory:

# ls /usr/lib64/nagios/plugins
eventhandlers negate urlize utils.sh

Sure enough, the usual suspects are not there. E.g.:

  • check_load
  • check_ping
  • check_disk
  • check_http
  • check_procs

Eventually I stumbled onto the following document, /usr/share/doc/nagios-plugins-2.0.3/README.Fedora:

Fedora users

Nagios plugins for Fedora have all been packaged separately. For
example, to isntall the check_http just install nagios-plugins-http.

All plugins are installed in the architecture dependent directory
/usr/lib{,64}/nagios/plugins/.

I installed some of the plugins following that convention:

sudo yum install nagios-plugins-load
sudo yum install nagios-plugins-ping
sudo yum install nagios-plugins-disk
sudo yum install nagios-plugins-http
sudo yum install nagios-plugins-procs

Now the the corresponding plugins exist in /usr/lib64/nagios/plugins, and Nagios reports OK for those checks on localhost.