Notifying a REST API from Icinga2

I wanted to send Icinga2 notifications to Slack. Some hosts and services don’t rise to the level of a PagerDuty notification, but e-mail just doesn’t cut it. A message in a Slack channel seemed an appropriate in-between.

This process is relatively straightforward, although I ran into some issues with SELinux that I will cover in this post.
Continue reading Notifying a REST API from Icinga2

cp, mv, ownership and attributes

I had always been under the impressions that when moving a file from one Linux filesystem to another (i.e. a new inode is created), that mv is essentially a cp command followed by an rm command.

That’s not quite correct. It is essentially a cp --archive command followed by an rm command.
Continue reading cp, mv, ownership and attributes

Icinga2 and PagerDuty integration

E-mail is not a good way to get my attention in a timely fashion. E-mail is inherently asynchronous, and healthy minds may ignore it for hours or even days at a time. So how do I handle monitoring alerts? One way is by using PagerDuty, a service that can call, text, or send push notifications to you (among other features).

I followed the steps at PagerDuty’s Icinga2 Integration Guide, but no alerts were coming through. What went wrong?
Continue reading Icinga2 and PagerDuty integration

SELinux, audit2why, audit2allow, and policy files

I’m no expert on SELinux, but I cringe whenever I read an online tutorial that includes the step Disable SELinux.

I ran into such a problem recently when I was installing Icinga. The service failed to start because of permissions issues creating the process ID (PID) file. One site suggested disabling SELinux, but I thought it was time to learn to update SELinux’s Type Enforcement (TE) policies instead.

First, I needed the audit2why tool, to explain what was being blocked and why:

# yum -q provides audit2why
policycoreutils-python-2.5-17.1.el7.x86_64 : SELinux policy core python
                                           : utilities
Repo        : base
Matched from:
Filename    : /usr/bin/audit2allow

I installed the policycoreutils-python package (and dependencies):

# yum install policycoreutils-python

I then ran audit2why against the audit log:

# audit2why -i /var/log/audit/audit.log
type=AVC msg=audit(1510711476.690:132): avc:  denied  { chown } for  pid=2459 comm="icinga" capability=0  scontext=system_u:system_r:nagios_t:s0 tcontext=system_u:system_r:nagios_t:s0 tclass=capability

        Was caused by:
                Missing type enforcement (TE) allow rule.

                You can use audit2allow to generate a loadable module to allow this access.

type=AVC msg=audit(1510711476.724:134): avc:  denied  { read write } for  pid=2465 comm="icinga" name="icinga.pid" dev="tmpfs" ino=19128 scontext=system_u:system_r:nagios_t:s0 tcontext=system_u:object_r:initrc_var_run_t:s0 tclass=file

        Was caused by:
                Missing type enforcement (TE) allow rule.

                You can use audit2allow to generate a loadable module to allow this access.

That’s still a little opaque. It’s not entirely clear to me why chown was blocked, for example. Look at the following specifics:

scontext=system_u:system_r:nagios_t:s0
tcontext=system_u:system_r:nagios_t:s0

To help decode that:

  • scontext = Source Context
  • tcontext = Target Context
  • _u:_r:_t:s# = user:role:type:security level

The source and target contexts are identical, and so it seems to me that the command should be allowed. But let’s try audit2allow and see what that tells us:

# audit2allow -i /var/log/audit/audit.log


#============= nagios_t ==============
allow nagios_t initrc_var_run_t:file { lock open read write };
allow nagios_t self:capability chown;

It is unclear to me how broad the first rule is: does it allow the nagios type (nagios_t) access to all initrc_var_run_t files? If so, that’s probably too broad. As the man page warns:

Care must be exercised while acting on the output of  this  utility  to
ensure  that  the  operations  being  permitted  do not pose a security
threat. Often it is better to define new domains and/or types, or  make
other structural changes to narrowly allow an optimal set of operations
to succeed, as opposed to  blindly  implementing  the  sometimes  broad
changes  recommended  by this utility.

That’s fairly terrifying. Although if the alternative is disabling SELinux completely, an overly broad SELinux policy is not the worst thing in the world.

So audit2allow provided a couple rules. Now what? Fortunately the audit2why and audit2allow man pages both include details on how to incorporate the rules into your SELinux policy. First, generate a new type enforcement policy:

# audit2allow -i /var/log/audit/audit.log --module local > local.te

This includes some extra information in addition to the default output:

# cat local.te

module local 1.0;

require {
        type nagios_t;
        type initrc_var_run_t;
        class capability chown;
        class file { lock open read write };
}

#============= nagios_t ==============
allow nagios_t initrc_var_run_t:file { lock open read write };
allow nagios_t self:capability chown;

Next the man page says:

# SELinux provides a policy devel environment under
# /usr/share/selinux/devel including all of the shipped
# interface files.
# You can create a te file and compile it by executing

$ make -f /usr/share/selinux/devel/Makefile local.pp

However, my system had no /usr/share/selinux/devel directory:

# ls /usr/share/selinux/
packages  targeted

I needed to install the policycoreutils-devel package (and dependencies):

# yum install policycoreutils-devel

Now compile the policy file to a binary:

# make -f /usr/share/selinux/devel/Makefile local.pp
Compiling targeted local module
/usr/bin/checkmodule:  loading policy configuration from tmp/local.tmp
/usr/bin/checkmodule:  policy configuration loaded
/usr/bin/checkmodule:  writing binary representation (version 17) to tmp/local.mod
Creating targeted local.pp policy package
rm tmp/local.mod.fc tmp/local.mod

Now install it using the semodule command:

# semodule -i local.pp

Did that solve the problem?

# systemctl start icinga
# systemctl status icinga
● icinga.service - LSB: start and stop Icinga monitoring daemon
   Loaded: loaded (/etc/rc.d/init.d/icinga; bad; vendor preset: disabled)
   Active: active (running) since Tue 2017-11-14 22:35:23 EST; 6s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 2661 ExecStop=/etc/rc.d/init.d/icinga stop (code=exited, status=0/SUCCESS)
  Process: 3838 ExecStart=/etc/rc.d/init.d/icinga start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/icinga.service
           └─3850 /usr/bin/icinga -d /etc/icinga/icinga.cfg

Nov 14 22:35:23 localhost.localdomain systemd[1]: Starting LSB: start and sto...
Nov 14 22:35:23 localhost.localdomain icinga[3838]: Running configuration che...
Nov 14 22:35:23 localhost.localdomain icinga[3838]: Icinga with PID  not runn...
Nov 14 22:35:23 localhost.localdomain icinga[3838]: Starting icinga: Starting...
Nov 14 22:35:23 localhost.localdomain systemd[1]: Started LSB: start and stop...
Nov 14 22:35:23 localhost.localdomain icinga[3850]: Finished daemonizing... (...
Nov 14 22:35:23 localhost.localdomain icinga[3850]: Event loop started...
Hint: Some lines were ellipsized, use -l to show in full.

It worked! The permissions issues were resolved without resorting to disabling SELinux.

There is still more I need to understand about SELinux, but it’s a start.

Additional reading:
CentOS: SELinux Policy Overview

Nagios check_disk returns DISK CRITICAL – /sys/kernel/config is not accessible: Permission denied

I enabled Nagios checks for free disk space on a group of servers today, and was hit with alerts containing the following error message:
DISK CRITICAL - /sys/kernel/config is not accessible: Permission denied

If you are looking for a solution, skip to the end. Some of my mistakes before finding the solution may be interesting though!

Continue reading Nagios check_disk returns DISK CRITICAL – /sys/kernel/config is not accessible: Permission denied