Icinga2 and PagerDuty integration

E-mail is not a good way to get my attention in a timely fashion. E-mail is inherently asynchronous, and healthy minds may ignore it for hours or even days at a time. So how do I handle monitoring alerts? One way is by using PagerDuty, a service that can call, text, or send push notifications to you (among other features).

I followed the steps at PagerDuty’s Icinga2 Integration Guide, but no alerts were coming through. What went wrong?

I checked the Icinga2 log file for messages containing pagerduty. On most systems:

grep -i pagerduty /var/log/icinga2/icinga2.log

It looked like a permissions issue:

[2018-09-07 16:50:20 -0500] warning/PluginNotificationTask: Notification command for object 'stephano' (PID: 11482, arguments: '/usr/local/bin/pagerduty_icinga.pl' 'enqueue' '-f' 'pd_nagios_object=host') terminated with exit code 128, output: execvpe(/usr/local/bin/pagerduty_icinga.pl flush) failed: Permission denied

What was going on?

I should note that I did not follow the instructions in the integration guide exactly. For example, I did not add the crontab entry to the icinga user’s crontab. I instead added the following to /etc/cron.d/pagerduty:

* * * * * icinga /usr/local/bin/pagerduty_icinga.pl flush

That should accomplish the thing, though.

Also, I made the permissions on /usr/local/bin/pagerduty_icinga.pl fairly restrictive, but the icinga user still had permission to read and execute the script:

$ ls -l /usr/local/bin/pagerduty_icinga.pl
-rwxr-x---. 1 root icinga 9144 Sep  7 16:18 /usr/local/bin/pagerduty_icinga.pl

Then I remembered to check SELinux:

$ sudo ausearch -f pagerduty_icinga.pl
type=AVC msg=audit(1541712215.916:326539): avc:  denied  { ioctl } for  pid=20609 comm="perl" path="/usr/local/bin/pagerduty_icinga.pl" dev="dm-2" ino=5529476 scontext=system_u:system_r:icinga2_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file

Sure enough, all of the other files in that directory had the context bin_t, but pagerduty_icinga.pl still had the SELinux type context from my home directory:

$ ls -lZ /usr/local/bin/pagerduty_icinga.pl
-rwxr-x---. root icinga unconfined_u:object_r:user_home_t:s0   /usr/local/bin/pagerduty_icinga.pl

I set the appropriate type context and ran restorecon:

$ sudo semanage fcontext -a -t bin_t /usr/local/bin/pagerduty_icinga.pl
$ sudo restorecon -v /usr/local/bin/pagerduty_icinga.pl
$ ls -lZ /usr/local/bin/pagerduty_icinga.pl
-rwxr-x---. root icinga unconfined_u:object_r:bin_t:s0   /usr/local/bin/pagerduty_icinga.pl

After that change, the PagerDuty integration worked!

The entire issue stemmed from the difference between copying [cp] the file (as specified in the integration guide) and moving [mv] the file. I figured there was no point in leaving a stray copy of the script in my home directory, so I simply moved the file:

$ sudo mv pagerduty_icinga.pl /usr/local/bin/

A copy of the file would have inherited the SELinux context of the parent directory (bin_t), but moving the file preserved the SELinux context.

As it turns out, mv includes an option to update the SELinux file context, which would have solved my problem:

-Z, --context
              set SELinux security context of destination file to default type

I have some additional thoughts about the differences between moving and copying files, but those will have to wait for another day.