{"id":3680,"date":"2023-02-12T19:47:47","date_gmt":"2023-02-13T00:47:47","guid":{"rendered":"https:\/\/osric.com\/chris\/accidental-developer\/?p=3680"},"modified":"2023-02-12T19:47:47","modified_gmt":"2023-02-13T00:47:47","slug":"running-splunk-in-aws","status":"publish","type":"post","link":"https:\/\/osric.com\/chris\/accidental-developer\/2023\/02\/running-splunk-in-aws\/","title":{"rendered":"Running Splunk in AWS"},"content":{"rendered":"

I don’t like using Google Analytics. The data is useful and well-presented, but I really just want basic web stats without sending all my web stats (along with data from my users) to Google. I’ve considered a number of other options, including Matomo<\/a>. But I already use Splunk at work, why not run Splunk at home too?<\/p>\n

Splunk Enterprise offers a 60-day trial license. After that, there’s a free license<\/a>. It’s not really clear that the free license covers what I’m trying to do here. The free license info includes:<\/p>\n

If you want to run Splunk Enterprise to practice searches, data ingestion, and other tasks without worrying about a license, Splunk Free is the tool for you.<\/p><\/blockquote>\n

I think this scenario qualifies. This is a hobby server. I use Splunk at my day job, so this is in some sense Splunk practice. I’ll give it more thought over the next 60 days. Your situation may vary!<\/p>\n

I launched an EC2 instance in AWS (Amazon Web Services). I picked a t2.micro instance. That instance size might be too small, but I’m not planning to send much data there. I picked Amazon Linux, which uses yum<\/code> and RPMs for package management, familiar from the RHEL, CentOS, and now Rocky Linux servers I use frequently. (One thing to note, the default user for Amazon Linux is ec2-user<\/code>. I always have to look that up.)<\/p>\n

For purposes of this post, I’ll use 203.0.113.18<\/code> as the EC2 instance’s public IP address. (203.0.113.0\/24<\/code> is an address block reserved for documentation, see RFC 5737<\/a>.)<\/p>\n

I transferred the RPM to the new server. I’m using Splunk 9.0.3, the current version as of this writing. I installed it:<\/p>\n

sudo yum install splunk-9.0.3-dd0128b1f8cd-linux-2.6-x86_64.rpm<\/code><\/pre>\n
Yum reported the installed size as 1.4 GB. Important to note, since I used an 8 GB HD, the default volume size when I launched the EC2 instance.<\/p>\n
I added an inbound rule the security group associated with the EC2 instance to allow 8000\/tcp traffic from my home IPv4 address.<\/p>\n
The installation works! I was able to connect to 203.0.113.18:8000<\/code> in a web browser. My connection to 203.0.113.18:8000<\/code> was not encrypted, but one thing at a time, right?<\/p>\n
Disk space, as I suspected, might be an issue. This warning appeared in Splunk’s health status:<\/p>\n
MinFreeSpace=5000. The diskspace remaining=3962 is less than 1 x minFreeSpace on \/opt\/splunk\/var\/lib\/splunk\/audit\/db<\/code><\/pre>\n
Next question: how do I get data into Splunk? The Splunk Enterprise download page helpfully includes a link to a “Getting Data In \u2014 Linux” video, although the video focused on ingesting local logs. I’m more interested in setting up the Splunk Universal Forwarder on a different server and ingesting logs from the osric.com web server. I installed the Splunk forwarder on the target web server.<\/p>\n
I enabled a receiver via Splunk web (see Enable a receiver for Splunk Enterprise<\/a> for information). I used the suggested port, 9997\/tcp.<\/p>\n
I also allowed this traffic from the web server’s IPv4 address via the AWS security group associated with the EC2 instance.<\/p>\n
I configured the forwarder on the target web server (see Configure the universal forwarder using configuration files<\/a> for more details):<\/p>\n
$ .\/bin\/splunk add forward-server 203.0.113.18:9997\r\nWarning: Attempting to revert the SPLUNK_HOME ownership\r\nWarning: Executing \"chown -R splunk \/opt\/splunkforwarder\"\r\nWARNING: Server Certificate Hostname Validation is disabled. Please see server.conf\/[sslConfig]\/cliVerifyServerName for details.\r\nSplunk username: admin\r\nPassword:\r\nAdded forwarding to: 203.0.113.18:9997.<\/code><\/pre>\n
I tried running a search, but the disk space limitations finally became apparent:<\/p>\n
Search not executed: The minimum free disk space (5000MB) reached for \/opt\/splunk\/var\/run\/splunk\/dispatch. user=admin., concurrency_category=\"historical\", concurrency_context=\"user_instance-wide\", current_concurrency=0, concurrency_limit=5000<\/code><\/pre>\n
I increased disk to 16 GB. (I’d never done that before for an EC2 instance, but it was surprisingly easy.)<\/p>\n
I needed to add something to monitor. On the target web server host I ran the following:<\/p>\n
$ sudo -u splunk \/opt\/splunkforwarder\/bin\/splunk add monitor \/var\/www\/chris\/data\/logs<\/code><\/pre>\n
The resulting output included the following message:<\/p>\n
Checking: \/opt\/splunkforwarder\/etc\/system\/default\/alert_actions.conf\r\n                Invalid key in stanza [webhook] in \/opt\/splunkforwarder\/etc\/system\/default\/alert_actions.conf, line 229: enable_allowlist (value: false).<\/code><\/pre>\n
It’s not clear if that’s actually a problem, and a few search results suggested it wasn’t worth worrying about.<\/p>\n
Everything was configured to forward data from the web server to Splunk. How could I find the data? I tried running a simple Splunk search:<\/p>\n
index=main<\/code><\/pre>\n
0 events returned. I also checked the indices at http:\/\/203.0.113.18:8000\/en-US\/manager\/search\/data\/indexes, which showed there were 0 events in the main<\/code> index.<\/p>\n
I ran tcpdump<\/code> on the target web server and confirmed there were successful connections to 203.0.113.18<\/code> on port 9997\/tcp:<\/p>\n
sudo tcpdump -i eth0 -nn port 9997<\/code><\/pre>\n
I tried another search on the Splunk web interface, this time querying some of Splunk’s internal indexes:<\/p>\n
index=_* osric<\/code><\/pre>\n
Several results were present. Clearly communication was happening. But where were the web logs?<\/p>\n
The splunk<\/code> user on the target web server doesn’t have permissions to read the web logs! I ran the following:<\/p>\n
chown apache:splunk \/var\/www\/chris\/data\/logs\/osric*<\/code><\/pre>\n
After that change, the Indexes page in the Splunk web interface still showed 0 events in the main index.<\/p>\n
I followed the advice on What are the basic troubleshooting steps in case of universal forwarder and heavy forwarder not forwarding data to Splunk?<\/a>, but still wasn’t seeing any issues. I took a close look again at the advice to check permissions. Tailing a specific log file worked fine, but getting a directory listing as the splunk<\/code> user failed:<\/p>\n
$ sudo -u splunk ls logs\r\nls: cannot open directory logs: Permission denied<\/code><\/pre>\n
Of course! The splunk<\/code> user had access to the logs themselves, but not to the directory containing them. It couldn’t enumerate the log files. I ran the following:<\/p>\n
$ sudo chgrp splunk logs<\/code><\/pre>\n
That did it! Logs were flowing! Search queries like the following produced results on the Splunk web interface:<\/p>\n
index=main<\/code><\/pre>\n
The search was slow, and there were warnings present when searching:<\/p>\n
Configuration initialization for \/opt\/splunk\/etc took longer than expected (1964ms) when dispatching a search with search ID 1676220274.309. This usually indicates problems with underlying storage performance.<\/code><\/pre>\n
I looks like t2.micro is much too small and under-powered for Splunk, even an instance with very little data (only 3 MB of data and 20,000 log events in the main index).<\/p>\n
Despite these drawbacks, the data was searchable. How did Splunk compare as a solution?<\/p>\n
Dashboards<\/strong>
\nI’ll need to create dashboards from scratch. I’ll want to know top pages, top URIs resulting in 404 errors, top user agents, etc. All of those will need to be built. It’s possible there’s a good Splunk app available that includes a lot of common dashboards for the Apache web server, but I haven’t really explored that.<\/p>\n
Google Analytics can’t report on 404 errors, but otherwise it provides a lot of comprehensive dashboards and data visualizations. Even if all you want are basic web stats, an application tailored to web analytics will include a lot of ready-made functionality.<\/p>\n
Robots, Spiders, and Crawlers (and More)<\/strong>
\nIt turns out, a large percentage of requests to my web server are not from human beings. Many of the requests are coming from robots. At least 31% of requests in the past day were coming from these 9 bots:<\/p>\n