I wanted to send Icinga2 notifications to Slack. Some hosts and services don’t rise to the level of a PagerDuty notification, but e-mail just doesn’t cut it. A message in a Slack channel seemed an appropriate in-between.<\/p>\n
This process is relatively straightforward, although I ran into some issues with SELinux that I will cover in this post.
\n<\/p>\n
First, create a Slack app at https:\/\/api.slack.com\/apps. I called mine Icinga2, and I installed it into my Slack workspace. Then activate Incoming Webhooks and add a new Webhook to the workspace. Since I was still very much testing this process, I selected a channel that I created for this purpose. Slack provides a unique webhook URI per app, per channel.<\/p>\n
Next, create a script to handle calling the webhook. Icinga2 includes scripts for notifying users via email:<\/p>\n
\/etc\/icinga2\/scripts\/mail-host-notification.sh<\/code><\/li>\n\/etc\/icinga2\/scripts\/mail-service-notification.sh<\/code><\/li>\n<\/ul>\nThose scripts are relatively complex and handle many options. For a proof-of-concept, I decided to create a new file that was much simpler:<\/p>\n
slack-host-notification.sh:<\/strong><\/code><\/p>\n#!\/bin\/sh\r\n\r\n# Environment variables\r\nNOTIFICATIONTYPE=$ICINGA_NOTIFICATIONTYPE\r\nHOSTDISPLAYNAME=$ICINGA_HOSTALIAS\r\nHOSTSTATE=$ICINGA_HOSTSTATE\r\nSLACK_WEBHOOK=$ICINGA_CONTACTPAGER\r\n\r\nMESSAGE=\"[$NOTIFICATIONTYPE] Host $HOSTDISPLAYNAME is $HOSTSTATE!\"\r\n\r\ncurl --silent \\\r\n --show-error \\\r\n -X POST \\\r\n -H \"Content-type: application\/json\" \\\r\n --data \"{\"text\":\"$MESSAGE\"}\" \\\r\n $SLACK_WEBHOOK<\/code><\/pre>\nDon’t forget to make it executable:<\/p>\n
sudo chmod +x \/etc\/icinga2\/scripts\/slack-host-notification.sh<\/code><\/pre>\nThe above script makes use of several environment variables, which will be set in the notification command below.<\/p>\n
Next, modify the Icinga2 configuration. This involves several Icinga2 objects:<\/p>\n
\n- User (corresponding to the Slack channel)<\/li>\n
- NotificationCommand (to call the
slack-host-notification.sh<\/code> script)<\/li>\n- Host (you could also modify an existing host)<\/li>\n<\/ul>\n
And then apply the Notification (to associate a notification command a host and user).<\/p>\n
\/etc\/icinga2\/conf.d\/slack.conf:<\/strong><\/code><\/p>\nobject User \"osric_net_slack\" {\r\n pager = \"<SLACK_WEBHOOK_URI>\"\r\n display_name = \"Slack Notifications User\"\r\n states = [ OK, Warning, Critical, Unknown, Up, Down ]\r\n types = [ Problem, Acknowledgement, Recovery ]\r\n}\r\n\r\nobject NotificationCommand \"slack-host-notification\" {\r\n command = [ SysconfDir + \"\/icinga2\/scripts\/slack-host-notification.sh\" ]\r\n env = {\r\n \"ICINGA_CONTACTPAGER\" = \"$user.pager$\"\r\n \"ICINGA_NOTIFICATIONTYPE\" = \"$notification.type$\"\r\n \"ICINGA_HOSTNAME\" = \"$host.name$\"\r\n \"ICINGA_HOSTALIAS\" = \"$host.display_name$\"\r\n \"ICINGA_HOSTSTATE\" = \"$host.state$\"\r\n \"ICINGA_HOSTOUTPUT\" = \"$host.output$\"\r\n }\r\n}\r\n\r\napply Notification \"slack-host\" to Host {\r\n command = \"slack-host-notification\"\r\n states = [ Up, Down ]\r\n types = [ Problem, Acknowledgement, Recovery ]\r\n period = \"24x7\"\r\n users = [ \"slack\" ]\r\n\r\n assign where host.vars.enable_slack == true\r\n}\r\n\r\nobject Host \"osric-web-007\" {\r\n import \"generic-host\"\r\n address = \"192.0.2.1\"\r\n vars.enable_slack = 1\r\n}<\/code><\/pre>\nI used an address in a range defined by RFC 5737<\/a> to be reserved for documentation, so it should, in theory, never actually exist. This is great for testing to see if a host is down<\/em>, but for the proof-of-concept I can’t test a recovery message. (Another method I have used for testing is to block outbound connections to a real host from the monitoring host via iptables to simulate an outage.)<\/p>\nAfter making these changes, be sure to validate the configuration before restarting:<\/p>\n
sudo \/usr\/sbin\/icinga2 daemon --validate<\/code><\/pre>\nIf everything works, restart the icinga2 service:<\/p>\n
sudo systemctl restart icinga2<\/code><\/pre>\nIf all goes well, the monitoring host should check the newly-added host, find it is unreachable, and send a notification to the Slack channel via the webhook.<\/p>\n
SELinux is enabled and in enforcing mode<\/strong><\/p>\nHowever, in my case, no notification appeared in Slack. I found an error in the logs.<\/p>\n
From \/var\/log\/icinga2\/icinga2.log<\/code>:<\/p>\n[2020-03-01 09:58:17 -0500] warning\/PluginNotificationTask: Notification command\r\n for checkable 'osric-web-007' and notification 'osric-web-007!slack-host' (PID:\r\n 16042, arguments: '\/etc\/icinga2\/scripts\/slack-host-notification.sh') terminated\r\n with exit code 128, output: execvpe(\/etc\/icinga2\/scripts\/slack-host-notification.sh) failed: Permission denied<\/code><\/pre>\nI double-checked the file mode, it was set to 0755. However, an audit log entry indicated it was due to mismatched SELinux types (from \/var\/log\/audit\/audit.log<\/code>):<\/p>\ntype=AVC msg=audit(1583074697.798:11573): avc: denied { execute } for pid=160\r\n42 comm=\"icinga2\" name=\"slack-host-notification.sh\" dev=\"dm-0\" ino=889294 sconte\r\nxt=system_u:system_r:icinga2_t:s0 tcontext=unconfined_u:object_r:icinga2_etc_t:s\r\n0 tclass=file permissive=0<\/code><\/pre>\nIf you have read a lot of AVC messages in the audit log, the above is fairly clear, but audit2why<\/code> can always help to clarify:<\/p>\nsudo audit2why -i \/var\/log\/audit\/audit.log:\r\n\r\ntype=AVC msg=audit(1583074697.798:11573): avc: denied { execute } for pid=16042 comm=\"icinga2\" name=\"slack-host-notification.sh\" dev=\"dm-0\" ino=889294 scontext=system_u:system_r:icinga2_t:s0 tcontext=unconfined_u:object_r:icinga2_etc_t:s0 tclass=file permissive=0\r\n\r\n Was caused by:\r\n Missing type enforcement (TE) allow rule.\r\n\r\n You can use audit2allow to generate a loadable module to allow this access.<\/code><\/pre>\naudit2allow<\/code> tries to be helpful, but in this case it isn’t really:<\/p>\n# audit2allow -i \/var\/log\/audit\/audit.log\r\n\r\n\r\n#============= icinga2_t ==============\r\nallow icinga2_t icinga2_etc_t:file execute;<\/code><\/pre>\nThat would technically<\/em> work, but there’s a better solution. Take a look at the SELinux file context on the notification scripts:<\/p>\nsudo ls -lZ \/etc\/icinga2\/scripts\r\n-rwxr-xr-x. root root system_u:object_r:nagios_notification_plugin_exec_t:s0 mail-host-notification.sh\r\n-rwxr-xr-x. root root system_u:object_r:nagios_notification_plugin_exec_t:s0 mail-service-notification.sh\r\n-rwxr-xr-x. root root unconfined_u:object_r:icinga2_etc_t:s0 slack-host-notification.sh<\/code><\/pre>\nThe mail-host-notification.sh<\/code> and mail-service-notification.sh<\/code> scripts both have an SELinux type context of nagios_notification_plugin_exec_t<\/code>. Use chcon<\/code> to change the SELinux context:<\/p>\nsudo chcon \\\r\n 'system_u:object_r:nagios_notification_plugin_exec_t:s0' \\\r\n \/etc\/icinga2\/scripts\/slack-host-notification.sh<\/code><\/pre>\nThat should solve it!<\/p>\n
A few additional notes on SELinux:<\/strong><\/p>\nOriginally, I had called curl<\/code> directly from the Icinga2 notification command (in slack.conf<\/code>).<\/p>\nThis resulted in a different SELinux error. Essentially, the Icinga2 process (with SELinux type context icinga2_t<\/code>) isn’t allowed to bind to any ports.<\/p>\nThere is an SELinux Boolean you can toggle that can change this:<\/p>\n
sudo setsebool -P icinga_can_connect_all 1<\/code><\/pre>\nHowever, allowing Icinga2 to connect to all ports seems unnecessary. I started looking at the source code for Icinga2’s SELinux policy:<\/p>\n
https:\/\/github.com\/Icinga\/icinga2\/blob\/master\/tools\/selinux\/icinga2.te<\/a><\/p>\nThat’s when I discovered some of the other type contexts included in the policy, including
\nnagios_notification_plugin_t<\/code>.<\/p>\n","protected":false},"excerpt":{"rendered":"I wanted to send Icinga2 notifications to Slack. Some hosts and services don’t rise to the level of a PagerDuty notification, but e-mail just doesn’t cut it. A message in a Slack channel seemed an appropriate in-between. This process is relatively straightforward, although I ran into some issues with SELinux that I will cover in … Continue reading Notifying a REST API from Icinga2<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[484],"tags":[495,217,346,458,544],"class_list":["post-3184","post","type-post","status-publish","format-standard","hentry","category-monitoring","tag-icinga2","tag-monitoring","tag-rest-api","tag-selinux","tag-slack"],"_links":{"self":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/3184","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/comments?post=3184"}],"version-history":[{"count":9,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/3184\/revisions"}],"predecessor-version":[{"id":3194,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/3184\/revisions\/3194"}],"wp:attachment":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/media?parent=3184"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/categories?post=3184"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/tags?post=3184"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}