NTP checks with icinga2

On my new Icinga2 monitoring host, I am slowly adding additional service checks to achieve parity with my existing Nagios monitoring. Next on my list, implementing NTP checks. The first step was to add a new service check to the Icinga2 configuration:

/etc/icinga2/conf.d/services.cfg:

apply Service "ntp_time" {
  import "generic-service"
  check_command = "ntp_time"
  assign where host.vars.os == "Linux"
}

The service check produced an error, as seen in the icingaweb2 interface:

execvpe(/usr/lib64/nagios/plugins/check_ntp_time) failed: No such file or directory

Oh! I don’t have the appropriate Nagios plugin installed on the Icinga2 host.

sudo yum install nagios-plugins-ntp

The NTP service check now reports OK on some hosts, but on other hosts I get a different error:

CRITICAL: No response from NTP server

The hosts that did not receive a response are all using chronyd. I edited /etc/chrony.conf and added:

allow 192.168.46.46

And restarted chronyd:

systemctl restart chronyd

Now all but one host reports OK. The last remaining host to show an error? The Icinga2 host itself!

allow 127.0.0.1

Another chronyd restart, and the NTP service on all hosts reports OK.

NRPE: Unable to read output

This one was a real facepalm moment, but I thought I’d share in case anyone else runs into the same thing.

I’ve been working on migrating from Nagios to Icinga2. One of the services I monitor is whether or not a given host has any available yum updates. This service, which I label check_yum, worked on all my hosts except for the Icinga2 host. All the other services monitored on that host were working, but check_yum returned an error:

NRPE: Unable to read output

I tried running the test manually on the Icinga2 host:

/usr/lib64/nagios/plugins/check_nrpe -H localhost -c check_yum
NRPE: Unable to read output

I checked to make sure NRPE was listening, in this case via xinetd:

lsof -i

I checked the service definition to see what script/plugin NRPE runs:

cat /etc/nrpe.d/check_yum.cfg
command[check_yum]=/usr/lib64/nagios/plugins/check_updates -w 0 -c 10 -t 60

I tried to run that manually and…the file /usr/lib64/nagios/plugins/check_updates did not exist.

I installed the corresponding yum package:

sudo yum install nagios-plugins-check-updates

Now it works! It was a reminder to myself to check the basics before trying to troubleshoot network issues.

Missing Nagios plugins in CentOS 7

I set up a Nagios server on a CentOS 7 VM (Virtual Machine):

sudo yum install epel-release
sudo yum install nrpe
sudo yum install nagios

By default it sets up some basic checks for localhost. When I checked the Nagios site at http://127.0.0.1/nagios/, I found that even PING was critical:

(No output on stdout) stderr: execvp(/usr/lib64/nagios/plugins/check_ping, ...) failed. errno is 2: No such file or directory

I checked the contents of the plugins directory:

# ls /usr/lib64/nagios/plugins
eventhandlers negate urlize utils.sh

Sure enough, the usual suspects are not there. E.g.:

  • check_load
  • check_ping
  • check_disk
  • check_http
  • check_procs

Eventually I stumbled onto the following document, /usr/share/doc/nagios-plugins-2.0.3/README.Fedora:

Fedora users

Nagios plugins for Fedora have all been packaged separately. For
example, to isntall the check_http just install nagios-plugins-http.

All plugins are installed in the architecture dependent directory
/usr/lib{,64}/nagios/plugins/.

I installed some of the plugins following that convention:

sudo yum install nagios-plugins-load
sudo yum install nagios-plugins-ping
sudo yum install nagios-plugins-disk
sudo yum install nagios-plugins-http
sudo yum install nagios-plugins-procs

Now the the corresponding plugins exist in /usr/lib64/nagios/plugins, and Nagios reports OK for those checks on localhost.