{"id":2762,"date":"2018-11-10T20:15:23","date_gmt":"2018-11-11T01:15:23","guid":{"rendered":"http:\/\/osric.com\/chris\/accidental-developer\/?p=2762"},"modified":"2018-11-10T20:15:23","modified_gmt":"2018-11-11T01:15:23","slug":"icinga2-and-pagerduty-integration","status":"publish","type":"post","link":"https:\/\/osric.com\/chris\/accidental-developer\/2018\/11\/icinga2-and-pagerduty-integration\/","title":{"rendered":"Icinga2 and PagerDuty integration"},"content":{"rendered":"

E-mail is not a good way to get my attention in a timely fashion. E-mail is inherently asynchronous, and healthy minds may ignore it for hours or even days at a time. So how do I handle monitoring alerts? One way is by using PagerDuty<\/a>, a service that can call, text, or send push notifications to you (among other features).<\/p>\n

I followed the steps at PagerDuty’s Icinga2 Integration Guide<\/a>, but no alerts were coming through. What went wrong?
\n<\/p>\n

I checked the Icinga2 log file for messages containing pagerduty<\/em>. On most systems:<\/p>\n

grep -i pagerduty \/var\/log\/icinga2\/icinga2.log<\/code><\/pre>\n
It looked like a permissions issue:<\/p>\n
[2018-09-07 16:50:20 -0500] warning\/PluginNotificationTask: Notification command for object 'stephano' (PID: 11482, arguments: '\/usr\/local\/bin\/pagerduty_icinga.pl' 'enqueue' '-f' 'pd_nagios_object=host') terminated with exit code 128, output: execvpe(\/usr\/local\/bin\/pagerduty_icinga.pl flush) failed: Permission denied<\/code><\/pre>\n
What was going on?<\/p>\n
I should note that I did not follow the instructions in the integration guide exactly. For example, I did not add the crontab entry to the icinga user’s crontab. I instead added the following to \/etc\/cron.d\/pagerduty<\/code>:<\/p>\n
* * * * * icinga \/usr\/local\/bin\/pagerduty_icinga.pl flush<\/code><\/pre>\n
That should accomplish the thing, though.<\/p>\n
Also, I made the permissions on \/usr\/local\/bin\/pagerduty_icinga.pl<\/code> fairly restrictive, but the icinga user still had permission to read and execute the script:<\/p>\n
$ ls -l \/usr\/local\/bin\/pagerduty_icinga.pl\r\n-rwxr-x---. 1 root icinga 9144 Sep  7 16:18 \/usr\/local\/bin\/pagerduty_icinga.pl<\/code><\/pre>\n
Then I remembered to check SELinux:<\/p>\n
$ sudo ausearch -f pagerduty_icinga.pl\r\ntype=AVC msg=audit(1541712215.916:326539): avc:  denied  { ioctl } for  pid=20609 comm=\"perl\" path=\"\/usr\/local\/bin\/pagerduty_icinga.pl\" dev=\"dm-2\" ino=5529476 scontext=system_u:system_r:icinga2_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file<\/code><\/pre>\n
Sure enough, all of the other files in that directory had the context bin_t<\/code>, but pagerduty_icinga.pl<\/code> still had the SELinux type context from my home directory:<\/p>\n
$ ls -lZ \/usr\/local\/bin\/pagerduty_icinga.pl\r\n-rwxr-x---. root icinga unconfined_u:object_r:user_home_t:s0   \/usr\/local\/bin\/pagerduty_icinga.pl<\/code><\/pre>\n
I set the appropriate type context and ran restorecon<\/code>:<\/p>\n
$ sudo semanage fcontext -a -t bin_t \/usr\/local\/bin\/pagerduty_icinga.pl\r\n$ sudo restorecon -v \/usr\/local\/bin\/pagerduty_icinga.pl\r\n$ ls -lZ \/usr\/local\/bin\/pagerduty_icinga.pl\r\n-rwxr-x---. root icinga unconfined_u:object_r:bin_t:s0   \/usr\/local\/bin\/pagerduty_icinga.pl<\/code><\/pre>\n
After that change, the PagerDuty integration worked!<\/p>\n
The entire issue stemmed from the difference between copying<\/em> [cp<\/code>] the file (as specified in the integration guide) and moving<\/em> [mv<\/code>] the file. I figured there was no point in leaving a stray copy of the script in my home directory, so I simply moved the file:<\/p>\n
$ sudo mv pagerduty_icinga.pl \/usr\/local\/bin\/<\/code><\/pre>\n
A copy of the file would have inherited the SELinux context of the parent directory (bin_t<\/code>), but moving the file preserved the SELinux context.<\/p>\n
As it turns out, mv<\/code> includes an option to update the SELinux file context, which would have solved my problem:<\/p>\n
-Z, --context\r\n              set SELinux security context of destination file to default type<\/code><\/pre>\n
I have some additional thoughts about the differences between moving and copying files, but those will have to wait for another day.<\/p>\n","protected":false},"excerpt":{"rendered":"
After following the steps in PagerDuty’s Icinga2 integration guide, I still wasn’t getting my expected PagerDuty alerts. This stemmed from a permissions issue relating to SELinux. I was able to update the SELinux type context for the PagerDuty script and then it worked as expected. This post describes the details of those steps.<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[484],"tags":[495,516,458],"class_list":["post-2762","post","type-post","status-publish","format-standard","hentry","category-monitoring","tag-icinga2","tag-pagerduty","tag-selinux"],"_links":{"self":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/2762","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/comments?post=2762"}],"version-history":[{"count":13,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/2762\/revisions"}],"predecessor-version":[{"id":2877,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/2762\/revisions\/2877"}],"wp:attachment":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/media?parent=2762"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/categories?post=2762"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/tags?post=2762"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}