{"id":2035,"date":"2017-09-09T00:22:47","date_gmt":"2017-09-09T05:22:47","guid":{"rendered":"http:\/\/osric.com\/chris\/accidental-developer\/?p=2035"},"modified":"2017-09-09T00:23:15","modified_gmt":"2017-09-09T05:23:15","slug":"nagios-check_disk-returns-disk-critical-sys-kernel-config-not-accessible-permission-denied","status":"publish","type":"post","link":"https:\/\/osric.com\/chris\/accidental-developer\/2017\/09\/nagios-check_disk-returns-disk-critical-sys-kernel-config-not-accessible-permission-denied\/","title":{"rendered":"Nagios check_disk returns DISK CRITICAL – \/sys\/kernel\/config is not accessible: Permission denied"},"content":{"rendered":"

I enabled Nagios checks for free disk space on a group of servers today, and was hit with alerts containing the following error message:
\nDISK CRITICAL - \/sys\/kernel\/config is not accessible: Permission denied<\/code><\/p>\n

If you are looking for a solution, skip to the end. Some of my mistakes before finding the solution may be interesting though!<\/p>\n

<\/p>\n

The wrong solution<\/strong><\/p>\n

Permission denied<\/em>? I had a hunch SELinux<\/a> was behind this. SELinux is behind every unexpected permissions problem lately. But before jumping to any conclusions, Google the error message. It’s rare to run across a problem that no one else has had before.<\/p>\n

Sure enough, I found a RedHat bug report<\/a> describing the same issue with the check_disk plugin. The developers closed the bug, saying that a new version has been released, and if it is still a problem someone should open a new bug for the new version. Initially I thought that was a terrible assumption. “We released a new version and have not confirmed whether or not this is still a bug. Therefore this isn’t a bug unless you do the work to re-report it.” Now that I have determined that it is not and likely never was a bug, I’m not sure I feel the same.<\/p>\n

The bug report mentions a workaround for the Nagios check_disk failure<\/a>. It is in fact a successful workaround. I don’t entirely like it, but assuming that only the root user can modify \/usr\/lib64\/nagios\/plugins\/check_disk<\/code> it seems like an acceptable risk. Still, recall that one of the benefits of SELinux is that even if another process owned by a user is compromised it doesn’t mean everything owned by that user gets compromised.<\/p>\n

Compare before and after:<\/p>\n

$ ls --context \/usr\/lib64\/nagios\/plugins\/check_disk\r\n-rwxr-xr-x. root root system_u:object_r:nagios_checkdisk_plugin_exec_t:s0 \/usr\/lib64\/nagios\/plugins\/check_disk\r\n$ sudo chcon -t nagios_unconfined_plugin_exec_t \/usr\/lib64\/nagios\/plugins\/check_disk\r\n$ ls --context \/usr\/lib64\/nagios\/plugins\/check_disk\r\n-rwxr-xr-x. root root system_u:object_r:nagios_unconfined_plugin_exec_t:s0 \/usr\/lib64\/nagios\/plugins\/check_disk<\/code><\/pre>\n
The SELinux type is now set to nagios_unconfined<\/strong>_plugin_exec_t<\/code>.<\/p>\n
Can I set the SELinux context via Ansible?<\/strong>
\nI need to make this change on several servers, and I need the change to be documented and repeatable, so I need to see how to best make that happen via Ansible.<\/p>\n
The Ansible sefcontext module<\/a> says it’s similar to the semanage fcontext<\/code> command, so it seemed like a good choice.<\/p>\n
First I tried the semange fcontext<\/code> command directly:<\/p>\n
$ sudo semanage fcontext -m -t nagios_unconfined_plugin_exec_t \/usr\/lib64\/nagios\/plugins\/check_disk\r\nValueError: File spec \/usr\/lib64\/nagios\/plugins\/check_disk conflicts with equivalency rule '\/usr\/lib64 \/usr\/lib'; Try adding '\/usr\/lib\/nagios\/plugins\/check_disk' instead\r\n$ sudo semanage fcontext -m -t nagios_unconfined_plugin_exec_t \/usr\/lib\/nagios\/plugins\/check_disk\r\nValueError: File context for \/usr\/lib\/nagios\/plugins\/check_disk is not defined\r\n\r\n$ sudo semanage fcontext --list \/usr\/lib64\/nagios\/plugins\/check_disk | grep check_disk\r\n\/usr\/lib\/nagios\/plugins\/check_disk                 regular file       system_u:object_r:nagios_checkdisk_plugin_exec_t:s0 \r\n\/usr\/lib\/nagios\/plugins\/check_disk_smb             regular file       system_u:object_r:nagios_checkdisk_plugin_exec_t:s0<\/code><\/pre>\n
The file looked like it had a defined context, didn’t it? A comment on the Fedora SELinux support list<\/a> had good advice:<\/p>\n
If the file context is not already defined in your local modification, you need to add is [sic], not modify<\/p><\/blockquote>\n
I tried again, adding<\/em> instead of modifying<\/em>, and comparing context before and after the change:<\/p>\n
$ ls --context \/usr\/lib64\/nagios\/plugins\/check_disk\r\n-rwxr-xr-x. root root system_u:object_r:nagios_checkdisk_plugin_exec_t:s0 \/usr\/lib64\/nagios\/plugins\/check_disk\r\n$ sudo semanage fcontext -a -t nagios_unconfined_plugin_exec_t \/usr\/lib\/nagios\/plugins\/check_disk\r\n$ sudo restorecon \/usr\/lib64\/nagios\/plugins\/check_disk\r\n$ ls --context \/usr\/lib64\/nagios\/plugins\/check_disk\r\n-rwxr-xr-x. root root system_u:object_r:nagios_unconfined_plugin_exec_t:s0 \/usr\/lib64\/nagios\/plugins\/check_disk<\/code><\/pre>\n
OK! That looks good. Now to do the same with Ansible. An excerpt of my Ansible role is below:<\/p>\n
- name: Allow Nagios to execute check_disk (change SELinux type)\r\n  sefcontext:\r\n    # Use lib, not lib64\r\n    # SELinux defines the equivalency rule '\/usr\/lib64 \/usr\/lib'\r\n    target: \/usr\/lib\/nagios\/plugins\/check_disk\r\n    setype: nagios_unconfined_plugin_exec_t\r\n    state: present<\/code><\/pre>\n
That didn’t work though. See the context returned below:<\/p>\n
$ ls --context \/usr\/lib64\/nagios\/plugins\/check_disk\r\n-rwxr-xr-x. root root system_u:object_r:nagios_checkdisk_plugin_exec_t:s0 \/usr\/lib64\/nagios\/plugins\/check_disk<\/code><\/pre>\n
Maybe it doesn’t run restorecon<\/code>? A thread on the Ansible Project Google Group<\/a> explains that “reload SELinux policy after commit” is not the same as restorecon<\/code>.<\/p>\n
The sefcontext module is roughly the functionality that ‘semanage fcontext’ provides you. It allows you to add SELinux file context mappings to the internal database.<\/p>\n
Now, the module is not intended to change file contexts based on the mapping, just like ‘semanage fcontext’ does not do. (See man semanage)<\/p>\n
As you said, you can do this with restorecon, or the file module….<\/p><\/blockquote>\n
The Ansible files module<\/a>! I gave it a try. Here’s an excerpt of my new Ansible role:<\/p>\n
- name: Allow Nagios to execute check_disk (change SELinux type)\r\n  file:\r\n    path: \/usr\/lib64\/nagios\/plugins\/check_disk\r\n    setype: nagios_unconfined_plugin_exec_t<\/code><\/pre>\n
Check the SELinux context:<\/p>\n
$ ls --context \/usr\/lib64\/nagios\/plugins\/check_disk\r\n-rwxr-xr-x. root root system_u:object_r:nagios_unconfined_plugin_exec_t:s0 \/usr\/lib64\/nagios\/plugins\/check_disk<\/code><\/pre>\n
That worked! That was easy! I use the Ansible files module all the time and didn’t even know that option was there!<\/p>\n
But wait. Have I just solved the wrong problem?<\/p>\n
The real solution<\/strong>
\nWhy does the error message say \/sys\/kernel\/config is not accessible<\/code>? That isn’t one of the disks in my system, is it?<\/p>\n
Turns out, it is. Run mount<\/code> to see all the filesystems. There are more of them than you might guess:<\/p>\n
$ mount | grep \/sys\/kernel\/config\r\nconfigfs on \/sys\/kernel\/config type configfs (rw,relatime)<\/code><\/pre>\n
What is configfs<\/code>?<\/p>\n
configfs is a ram-based filesystem that…is a filesystem-based manager of kernel objects<\/p><\/blockquote>\n
(from configfs.txt<\/a>)<\/p>\n
That’s the real problem. I’m checking a filesystem that I didn’t intend to, and one that Nagios probably shouldn’t have access to.<\/p>\n
I checked the default config in \/etc\/nagios\/nrpe.cfg<\/code> and found these (the latter is commented-out by default, as shown below):<\/p>\n
command[check_hda1]=\/usr\/lib64\/nagios\/plugins\/check_disk -w 20% -c 10% -p \/dev\/hda1\r\n#command[check_disk]=\/usr\/lib64\/nagios\/plugins\/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$<\/code><\/pre>\n
My config had a custom definition in a cfg file in \/etc\/nrpe.d<\/code>:<\/p>\n
command[check_disk]=\/usr\/lib64\/nagios\/plugins\/check_disk -w 10 -c 5 -X devfs -X$<\/code><\/pre>\n
The above excludes devfs<\/code> filesystems, which don’t even exist on my CentOS 7 hosts. The -X$<\/code> looks like a mistake, possibly a copy-paste error of a line truncated by the terminal.<\/p>\n
I checked and confirmed that even without the bad exclusions, the check_disk command still produced the same error:<\/p>\n
command[check_disk]=\/usr\/lib64\/nagios\/plugins\/check_disk -w 10 -c 5<\/code><\/pre>\n
I created a revised definition that excludes the problematic filesystem:<\/p>\n
command[check_disk]=\/usr\/lib64\/nagios\/plugins\/check_disk -w 10% -c 5% -X configfs<\/code><\/pre>\n
It works! No more errors.<\/p>\n
It might be better, of course, to include <\/em>only the filesystems I expect to be checking:<\/p>\n
command[check_disk]=\/usr\/lib64\/nagios\/plugins\/check_disk -w 10% -c 5% -N xfs<\/code><\/pre>\n
The above also works.<\/p>\n
Be sure to see the check_disk plugin docs<\/a> for the full list of parameters. It isn’t completely obvious, but the second example implies that, unless otherwise specified, check_disk<\/code> tries to check all<\/em> available filesystems.<\/p>\n
Summary<\/strong><\/p>\n
    \n
  1. Read the error messages.<\/li>\n
  2. Understand the error messages.<\/li>\n
  3. Google the error messages, but the results don’t mean much if no one else read and understood the error messages.<\/li>\n
  4. Don’t change SELinux settings unless absolutely necessary. The defaults are there for a reason.<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"
    The Nagios check_disk plugin returned the error “DISK CRITICAL – \/sys\/kernel\/config is not accessible: Permission denied”. At first I blamed SELinux and tried to change the SELinux context, but in reality I was using a bad Nagios command definition.<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[439,422],"tags":[423,348,458],"class_list":["post-2035","post","type-post","status-publish","format-standard","hentry","category-ansible","category-sysadmin","tag-ansible","tag-nagios","tag-selinux"],"_links":{"self":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/2035","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/comments?post=2035"}],"version-history":[{"count":18,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/2035\/revisions"}],"predecessor-version":[{"id":2053,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/2035\/revisions\/2053"}],"wp:attachment":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/media?parent=2035"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/categories?post=2035"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/tags?post=2035"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}