check_http returns 403 Forbidden on fresh Nagios installation

I recently installed a Nagios server on a new CentOS 7 virtual machine (on Virtual Box).

One of the default checks included upon installation is a check on localhost to confirm that the HTTP server is responding. (First I had to install the check_http plugin, see previous post.) The Nagios web interface reports a warning for this check:

HTTP WARNING: HTTP/1.1 403 Forbidden - 5261 bytes in 0.001 second response time

This is unexpected, since I can request the same page in a browser, which returns the Apache Welcome page.

When I run the check manually I get the same result, as expected:

# /usr/lib64/nagios/plugins/check_http -H localhost
HTTP WARNING: HTTP/1.1 403 Forbidden - 5261 bytes in 0.001 second response time |time=0.000907s|;;;0.000000 size 5261B;;;0

I checked with curl:

# curl http://localhost

This returns the HTML source of the Apache Welcome page. It looks like it is working, right? But looking at the headers returned by the Apache server also shows 403 Forbidden:

# curl -I http://localhost
HTTP/1.1 403 Forbidden
...

The Apache Welcome page gives some hints about this behavior:

Are you the Administrator?

You should add your website content to the directory /var/www/html/.

To prevent this page from ever being used, follow the instructions in the file /etc/httpd/conf.d/welcome.conf.

The /etc/httpd/conf.d/welcome.conf file begins with the following comments and directive:

#
# This configuration file enables the default "Welcome" page if there
# is no default index page present for the root URL.  To disable the
# Welcome page, comment out all the lines below.
#
# NOTE: if this file is removed, it will be restored on upgrades.
#
<LocationMatch "^/+$">
    Options -Indexes
    ErrorDocument 403 /.noindex.html
</LocationMatch>

The Apache config is specifying that if there is no index page for the document root, return the Welcome page as an error document with a 403 HTTP status code.

Once I added a basic HTML file at /var/www/html/index.html, Nagios returned a success message:

HTTP OK: HTTP/1.1 200 OK - 549 bytes in 0.001 second response time

Applying per directory X-Frame-Options headers in Apache

To help prevent against click-jacking, I had applied the following to my Apache 2.2 configuration based on the suggestions described in OWASP’s Clickjacking Defense Cheat Sheet and Mozilla Developer Network’s The X-Frame-Options response header:

Header always append X-Frame-Options SAMEORIGIN

However, my site has certain pages that are included in an iframe on another site, for the purpose of displaying content on digital signage devices. After I added that header, those pages would no longer load in an iframe on the digital signage devices’ browsers.

I thought I might be able to change SAMEORIGIN to ALLOW-FROM and list both the URI of my site and the URI of the digital signage page. However, the HTTP Header Field X-Frame-Options RFC indicates:

Wildcards or lists to declare multiple domains in one ALLOW-FROM statement are not permitted

The pages I wanted to exempt from the X-Frame-Options restriction exist in their own directory, /digitalsignage, so I tried to override the X-Frame-Options header in a .htaccess file:

Header always append X-Frame-Options ALLOW-ACCESS http://example.com

That caused a 500 Server Error. This message appeared in the error logs:

.htaccess: error: envclause should be in the form env=envar

The Header directive must be malformed, but I’m am not sure how. I did not determine how to properly format the statement so as not to produce that error, although several sites have pointed out that some browsers (Chrome, Safari) do not support ALLOW-ACCESS.

I changed the .htaccess file back to SAMEORIGIN, to match what was in the main site configuration:

Header always append X-Frame-Options SAMEORIGIN

I then noted that the response header sent by the server included SAMEORIGIN twice:

Header: SAMEORIGIN, SAMEORIGIN

That’s the expected behavior when using append. It appeared only once after I changed append to set:
Header always set X-Frame-Options SAMEORIGIN

I tried using set instead of append with ALLOW-ACCESS:

Header always set X-Frame-Options ALLOW-ACCESS http://example.com

But it still produced the same 500 Server Error.

After reading the documentation for Apache’s mod_headers, I realized that unset would allow me to remove the X-Frame-Options header from the /digitalsignage directory:
Header always unset X-Frame-Options

That worked, and the pages were successfully included as iframes in a page on the digital signage company’s site.

Monitoring web server status with a shell script

Recently, my VPS (Virtual Private Server) ran into some issues where it exceeded the maximum amount of RAM allotted under my subscription. When this happens, the web server software shuts down and does not restart until I manually restart it.

This is bad. I’m not always visiting my own web site, so it could be down for days without me knowing. Although I really need to identify what is using all the RAM, in the meantime I’ll settle for a monitoring system that will notify me when the server is down.

#!/bin/bash
if curl -s --head http://osric.com/ | grep "200 OK" > /dev/null
  then 
    echo "The HTTP server on osric.com is up!" > /dev/null
  else
    echo "The HTTP server on osric.com is down!"
fi

cURL will let you retrieve a URL via the command line, and provides more options than Wget for a single URL. In this case, I used the silent switch to eliminate the status/progress output, and the head switch to retrieve only the document headers. The document header is then piped to Grep, which searches for the string “200 OK” (the HTTP status message for a successful request).

I send the result of that to /dev/null so that the output doesn’t appear on the screen.

If grep does find 200 OK, then I send a success message to /dev/null. This is largely unnecessary, but it is nice to leave in to test the script in a successful case–just remove the > /dev/null. If it doesn’t find 200 OK, then there is a problem. It might not mean, necessarily, that the web server is down, but it definitely indicates there is a problem that needs to be identified.

I added a call to this script to a crontab to run every 5 minutes. If there is no output, nothing happens. If there is output, the output is sent to me via e-mail, which, assuming I am checking my e-mail religiously, should reduce server downtime.

Apache Install and Ambiguous Errors

I installed Apache 2.2.11 on the Windows XP portion of my desktop workstation for development purposes, but I got a lot of ambiguous errors when starting from the Apache Service Monitor or the Windows start menu.

Finally, when I started Apache from the command line I got a more informative error:
(OS 10048) Only one usage of each socket address (protocal/network address/port) is normally permitted. : make_sock: could not bind to address 127.0.0.1:80 no listening sockets available, shutting down

It turns out, I had Skype running, which by default binds to ports 5520, 80, and 443. There are several solutions:
Continue reading Apache Install and Ambiguous Errors