Notifying a REST API from Icinga2

I wanted to send Icinga2 notifications to Slack. Some hosts and services don’t rise to the level of a PagerDuty notification, but e-mail just doesn’t cut it. A message in a Slack channel seemed an appropriate in-between.

This process is relatively straightforward, although I ran into some issues with SELinux that I will cover in this post.

First, create a Slack app at https://api.slack.com/apps. I called mine Icinga2, and I installed it into my Slack workspace. Then activate Incoming Webhooks and add a new Webhook to the workspace. Since I was still very much testing this process, I selected a channel that I created for this purpose. Slack provides a unique webhook URI per app, per channel.

Next, create a script to handle calling the webhook. Icinga2 includes scripts for notifying users via email:

  • /etc/icinga2/scripts/mail-host-notification.sh
  • /etc/icinga2/scripts/mail-service-notification.sh

Those scripts are relatively complex and handle many options. For a proof-of-concept, I decided to create a new file that was much simpler:

slack-host-notification.sh:

#!/bin/sh

# Environment variables
NOTIFICATIONTYPE=$ICINGA_NOTIFICATIONTYPE
HOSTDISPLAYNAME=$ICINGA_HOSTALIAS
HOSTSTATE=$ICINGA_HOSTSTATE
SLACK_WEBHOOK=$ICINGA_CONTACTPAGER

MESSAGE="[$NOTIFICATIONTYPE] Host $HOSTDISPLAYNAME is $HOSTSTATE!"

curl --silent \
     --show-error \
     -X POST \
     -H "Content-type: application/json" \
     --data "{"text":"$MESSAGE"}" \
     $SLACK_WEBHOOK

Don’t forget to make it executable:

sudo chmod +x /etc/icinga2/scripts/slack-host-notification.sh

The above script makes use of several environment variables, which will be set in the notification command below.

Next, modify the Icinga2 configuration. This involves several Icinga2 objects:

  • User (corresponding to the Slack channel)
  • NotificationCommand (to call the slack-host-notification.sh script)
  • Host (you could also modify an existing host)

And then apply the Notification (to associate a notification command a host and user).

/etc/icinga2/conf.d/slack.conf:

object User "osric_net_slack" {
  pager = "<SLACK_WEBHOOK_URI>"
  display_name = "Slack Notifications User"
  states = [ OK, Warning, Critical, Unknown, Up, Down ]
  types = [ Problem, Acknowledgement, Recovery ]
}

object NotificationCommand "slack-host-notification" {
  command = [ SysconfDir + "/icinga2/scripts/slack-host-notification.sh" ]
  env = {
    "ICINGA_CONTACTPAGER" = "$user.pager$"
    "ICINGA_NOTIFICATIONTYPE" = "$notification.type$"
    "ICINGA_HOSTNAME" = "$host.name$"
    "ICINGA_HOSTALIAS" = "$host.display_name$"
    "ICINGA_HOSTSTATE" = "$host.state$"
    "ICINGA_HOSTOUTPUT" = "$host.output$"
  }
}

apply Notification "slack-host" to Host {
  command = "slack-host-notification"
  states = [ Up, Down ]
  types = [ Problem, Acknowledgement, Recovery ]
  period = "24x7"
  users = [ "slack" ]

  assign where host.vars.enable_slack == true
}

object Host "osric-web-007" {
  import "generic-host"
  address = "192.0.2.1"
  vars.enable_slack = 1
}

I used an address in a range defined by RFC 5737 to be reserved for documentation, so it should, in theory, never actually exist. This is great for testing to see if a host is down, but for the proof-of-concept I can’t test a recovery message. (Another method I have used for testing is to block outbound connections to a real host from the monitoring host via iptables to simulate an outage.)

After making these changes, be sure to validate the configuration before restarting:

sudo /usr/sbin/icinga2 daemon --validate

If everything works, restart the icinga2 service:

sudo systemctl restart icinga2

If all goes well, the monitoring host should check the newly-added host, find it is unreachable, and send a notification to the Slack channel via the webhook.

SELinux is enabled and in enforcing mode

However, in my case, no notification appeared in Slack. I found an error in the logs.

From /var/log/icinga2/icinga2.log:

[2020-03-01 09:58:17 -0500] warning/PluginNotificationTask: Notification command
 for checkable 'osric-web-007' and notification 'osric-web-007!slack-host' (PID:
 16042, arguments: '/etc/icinga2/scripts/slack-host-notification.sh') terminated
 with exit code 128, output: execvpe(/etc/icinga2/scripts/slack-host-notification.sh) failed: Permission denied

I double-checked the file mode, it was set to 0755. However, an audit log entry indicated it was due to mismatched SELinux types (from /var/log/audit/audit.log):

type=AVC msg=audit(1583074697.798:11573): avc:  denied  { execute } for  pid=160
42 comm="icinga2" name="slack-host-notification.sh" dev="dm-0" ino=889294 sconte
xt=system_u:system_r:icinga2_t:s0 tcontext=unconfined_u:object_r:icinga2_etc_t:s
0 tclass=file permissive=0

If you have read a lot of AVC messages in the audit log, the above is fairly clear, but audit2why can always help to clarify:

sudo audit2why -i /var/log/audit/audit.log:

type=AVC msg=audit(1583074697.798:11573): avc:  denied  { execute } for  pid=16042 comm="icinga2" name="slack-host-notification.sh" dev="dm-0" ino=889294 scontext=system_u:system_r:icinga2_t:s0 tcontext=unconfined_u:object_r:icinga2_etc_t:s0 tclass=file permissive=0

        Was caused by:
                Missing type enforcement (TE) allow rule.

                You can use audit2allow to generate a loadable module to allow this access.

audit2allow tries to be helpful, but in this case it isn’t really:

# audit2allow -i /var/log/audit/audit.log


#============= icinga2_t ==============
allow icinga2_t icinga2_etc_t:file execute;

That would technically work, but there’s a better solution. Take a look at the SELinux file context on the notification scripts:

sudo ls -lZ /etc/icinga2/scripts
-rwxr-xr-x. root root system_u:object_r:nagios_notification_plugin_exec_t:s0 mail-host-notification.sh
-rwxr-xr-x. root root system_u:object_r:nagios_notification_plugin_exec_t:s0 mail-service-notification.sh
-rwxr-xr-x. root root unconfined_u:object_r:icinga2_etc_t:s0 slack-host-notification.sh

The mail-host-notification.sh and mail-service-notification.sh scripts both have an SELinux type context of nagios_notification_plugin_exec_t. Use chcon to change the SELinux context:

sudo chcon \
 'system_u:object_r:nagios_notification_plugin_exec_t:s0' \
 /etc/icinga2/scripts/slack-host-notification.sh

That should solve it!

A few additional notes on SELinux:

Originally, I had called curl directly from the Icinga2 notification command (in slack.conf).

This resulted in a different SELinux error. Essentially, the Icinga2 process (with SELinux type context icinga2_t) isn’t allowed to bind to any ports.

There is an SELinux Boolean you can toggle that can change this:

sudo setsebool -P icinga_can_connect_all 1

However, allowing Icinga2 to connect to all ports seems unnecessary. I started looking at the source code for Icinga2’s SELinux policy:

https://github.com/Icinga/icinga2/blob/master/tools/selinux/icinga2.te

That’s when I discovered some of the other type contexts included in the policy, including
nagios_notification_plugin_t.

Leave a Reply

Your email address will not be published. Required fields are marked *