Category: security

IPBan’s ProcessToRunOnBan functionality

The IPBan config file contains 2 interesting items that can trigger actions when IP addresses are banned or unbanned: ProcessToRunOnBan and ProcessToRunOnUnban.

Here’s the default config entry for ProcessToRunOnBan:


<add key="ProcessToRunOnBan" value=""/>

I decided I wanted to make the list of banned IP addresses public by writing to a web-accessible file. I tried adding the following values:


<add key="ProcessToRunOnBan" value="%SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe C:\add-bannedip.ps1|###IPADDRESS###"/>

<add key="ProcessToRunOnUnban" value="%SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe C:\remove-unbannedip.ps1|###IPADDRESS###"/>

The PowerShell (.ps1) scripts were really simple. The first adds banned IPs to a text file within the web root:

# Add an IP address to the list of banned IPs
param ($BannedIP)

$BannedPath = 'C:\inetpub\wwwroot\banned.txt'
Add-Content -Path $BannedPath -Value $BannedIP

The next removes unbanned IPs from the same text file:

# Remove an IP address from the list of banned IPs
param ($UnbannedIP)

$BannedPath = 'C:\inetpub\wwwroot\banned.txt'
Set-Content -Path $BannedPath (Get-Content $BannedPath | Select-String -NotMatch $UnbannedIP)

There are some flaws and a lack of error-checking in the above. The un-ban script could match IP addresses that are not identical, for example: 192.0.2.1 would match 192.0.2.10 and 192.0.2.100. Additionally, I would want to confirm that the parameter value was a valid IP address, but this was just a quick proof-of-concept.

However, I encountered an error when the next IP address was banned:

2022-01-03 00:51:35.8763|ERROR|DigitalRuby.IPBanCore.Logger|Failed to execute process C:\Program Files\IPBan\%SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe C:\add-bannedip.ps1 192.0.2.14: System.ComponentModel.Win32Exception (2): An error occurred trying to start process 'C:\Program Files\IPBan\%SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe C:\add-bannedip.ps1' with working directory 'C:\Program Files\IPBan\%SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe C:'. The system cannot find the file specified.
   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo )
   at System.Diagnostics.Process.StartCore(ProcessStartInfo )
   at System.Diagnostics.Process.Start()
   at System.Diagnostics.Process.Start(ProcessStartInfo )
   at DigitalRuby.IPBanCore.IPBanService.<>c__DisplayClass191_0.<ExecuteExternalProcessForIPAddresses>b__0() in C:\Users\Jeff\Documents\GitHub\DigitalRuby\IPBan\IPBanCore\Core\IPBan\IPBanService_Private.cs:line 591

It apparently failed to expand %SystemRoot%, so I replaced it with C:\Windows.

As mentioned in my previous post on IPBan (IPBan: fail2ban for Windows), I am using a remote config file hosted on an HTTPS server. Shortly after I made the change on the remote server, I noticed this in the logs (logfile.txt):

2022-01-03 01:36:43.3348|INFO|DigitalRuby.IPBanCore.Logger|Config file changed

It looks like IPBan automatically checks the GetUrlConfig value for updates. I confirmed that the file at C:\Program Files\IPBan\ipban.config was updated at the same time. This is excellent, previously I thought I might need to restart the IPBan service any time the configuration changed.

Unfortunately, my change still didn’t work. I encountered the following error:

2022-01-03 14:22:21.7154|ERROR|DigitalRuby.IPBanCore.Logger|Failed to execute process C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe C:\add-bannedip.ps1 192.0.2.208: System.ComponentModel.Win32Exception (2): An error occurred trying to start process 'C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe C:\add-bannedip.ps1' with working directory 'C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe C:'. The system cannot find the file specified.
   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo )
   at System.Diagnostics.Process.Start(ProcessStartInfo )
   at DigitalRuby.IPBanCore.IPBanService.<>c__DisplayClass191_0.<ExecuteExternalProcessForIPAddresses>b__0() in C:\Users\Jeff\Documents\GitHub\DigitalRuby\IPBan\IPBanCore\Core\IPBan\IPBanService_Private.cs:line 591

I decided to take a closer look at line 591:

    ProcessStartInfo psi = new()
    {
        FileName = programFullPath,
        WorkingDirectory = Path.GetDirectoryName(programFullPath),
        Arguments = replacedArgs
    };
    using Process p = Process.Start(psi);

It looked to me like it was taking the path for the PowerShell executable and the PowerShell script both as the full path. I changed the config to pass in the path to the PowerShell script as part of the arguments, which made sense:

<add key="ProcessToRunOnBan" value="C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe|C:\add-bannedip.ps1 ###IPADDRESS###"/>

That worked! The next time an IP address was banned, the .ps1 script was run and added the IP address to a web-accessible file.

IPBan: fail2ban for Windows

I was looking for a tool to block IP addresses after a certain number of failed RDP login attempts, something like fail2ban but for Windows. I came across IPBan. Calling IPBan a “fail2ban for Windows” unfairly minimizes what it can do, but it can handle that task quite nicely.

As a test I installed it on a Windows 2019 Server running in Azure using the provided install instructions from the README:

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; iex ((New-Object System.Net.WebClient).DownloadString('https://raw.githubusercontent.com/DigitalRuby/IPBan/master/IPBanCore/Windows/Scripts/install_latest.ps1'))

You don’t want to blindly run a script that downloads and installs something on your machine without looking at it first, do you? Of course not. You take a look at the code first:

https://github.com/DigitalRuby/IPBan/blob/master/IPBanCore/Windows/Scripts/install_latest.ps1

Essentially, the script does the following:

  1. Downloads a zip file
  2. Extracts the files
  3. Sets up a Windows service
  4. Suggests that you might want to edit the config by opening the config file in Notepad

I took a look at the config file and made only one change to get started: I added a CenturyLink subnet to the allow list. CenturyLink is my ISP, and I didn’t want to accidentally lock myself out. Normally you wouldn’t want to add an entire /12 of your residential ISP, but this was just a test on a temporary and disposable server:

<add key="Whitelist" value="75.160.0.0/12"/>

I noted a few other interesting things in the config file:

  1. It can also examine Linux logs (IPBan can run on Windows or Linux)
  2. It blocks failed RDP logins, but also blocks failed logins for other Windows services, such as MSSQL and Exchange
  3. It defaults to banning an IP address after 5 failed attempts, but the number of failed attempts can be configured
  4. The default ban duration is 1 day, but this can be configured
  5. The default ban duration can take into account recidivism, so that the second or third time an address is banned it can use a longer duration
  6. It can also use a username-based allow list (whitelist), so that attempted logins from usernames not on the list will ban the IP address immediately
  7. It can add blocks based on external threat intelligence feeds, and uses the Emerging Threats blocklist by default
  8. It can pull the config file from a URL, allowing you to manage the configuration of many servers from one location.

After I reviewed the config file (C:\Program Files\IPBan\ipban.config) I restarted the IPBAN service via Services GUI. You could also restart it via Powershell:

sc.exe stop IPBAN
sc.exe start IPBAN

Now that it was up-and-running, what changes did it make? I opened up Windows Defender Firewall and opened the Advanced Settings. Looking though the Inbound rules I found 4 rules prefixed with IPBAN_, three Deny rules and one Allow rule:

  1. [Deny] IPBan_Block_0
  2. [Deny] IPBan_EmergingThreats_0
  3. [Deny] IPBan_EmergingThreats_1000
  4. [Allow] IPBan_GlobalWhitelist_0

I looked at the properties for IPBan_Block_0. Already, one IP address had been banned! (I checked several hours later and only 2 additional IP addresses had been banned, so I may have gotten lucky to see one within minutes.)

It took me a while to figure out what the difference between IPBan_EmergingThreats_0 and IPBan_EmergingThreats_1000. IPBan is creating a new firewall rule after 999 entries, so the 1000th IP address or subnet from the Emerging Threats feed was added to IPBan_EmergingThreats_1000. I’ve seen some arguments online about whether or not there is a limit to the number of addresses or subnets that can be included in the scope for a Windows Defender Firewall rule, and some sources indicate there is a limit of 1000 (and I’m fairly certain the author of IPBan is one of the people arguing there is a limit).

The IPBan_GlobalWhitelist_0 contained the CenturyLink subnet that I explicitly allowed.

I was excited about pulling the configuration from a URL, so I added my configuration to https://osric.com/chris/ipban/ipban.example.config, initially with just one tweak:

<add key="UserNameWhitelist" value="chris"/>

This would, in theory, ban any IP address immediately if they used an address like admin or guest. It uses a configurable Levenshtein distance to try to avoid banning an IP address based on a typo (for example, chros instead of chris), which is a clever approach.

I then added the URL to the config file on the server itself:

<add key="GetUrlConfig" value="https://osric.com/chris/ipban/ipban.example.config"/>

After another service restart, I wanted to test to see if a login attempt with a bad username would cause an immediate block. I opened an RDP connection using the credentials admin:admin and from a VPN IP address that would be outside of the allowed CenturyLink subnet.

I did not immediately see the VPN IP address in the IPBAN_Block_0 deny rule. I checked the logs (C:\Program Files\IPBan\logfile.txt):

2021-12-31 04:35:45.3336|INFO|DigitalRuby.IPBanCore.Logger|Firewall entries updated: 
2021-12-31 04:37:45.5791|WARN|DigitalRuby.IPBanCore.Logger|Login failure: 198.51.100.101, admin, RDP, 1
2021-12-31 04:37:45.5791|INFO|DigitalRuby.IPBanCore.Logger|IP blacklisted: False, user name blacklisted: False, fails user name white list regex: False, user name edit distance blacklisted: True
2021-12-31 04:37:45.5791|WARN|DigitalRuby.IPBanCore.Logger|Banning ip address: 198.51.100.101, user name: admin, config black listed: True, count: 1, extra info: , duration: 1.00:00:00
2021-12-31 04:37:45.6024|WARN|DigitalRuby.IPBanCore.Logger|Updating firewall with 1 entries...
2021-12-31 04:37:45.6024|INFO|DigitalRuby.IPBanCore.Logger|Firewall entries updated: 198.51.100.101

There was just a delay in adding it. I checked the IPBan_Block_0 deny rule again in Windows Defender Firewall and the IP address was there. I must have been exceedingly quick, as the cycle time defaults to 15 seconds:

<add key="CycleTime" value="00:00:00:15"/>

One other change I made to my config: I set this option to false so that any future tests would not send my VPN IP to the “global ipban database”:

<add key="UseDefaultBannedIPAddressHandler" value="true" />

Normally leaving that set to true should be fine, but during testing it would be good to avoid adding your own IP address to a block list. I have not yet discovered if the global IPBan database is publicly available.

A couple other things I noted:

The ipban.config file was completely overwritten by the version at the GetUrlConfig, including the GetUrlConfig value! The next time I restarted the IPBAN service, it did not pick up the config from that URL, as the GetUrlConfig option was now blank. I updated the GetUrlConfig value on ipban.config locally and on the remotely hosted ipban.example.config to include the URL to itself, so that it would persist after a restart.

The FirewallRules config option has a somewhat confusing syntax, at least for Deny rules. I added the following rule, which then appeared in my Windows Defender Firewalls block rules as IPBan_EXTRA_GoogleDNS_0:

<add key="FirewallRules" value="
    GoogleDNS;block;8.8.4.4;53;.
"/>

This rule allows inbound port 53/tcp traffic from 8.8.4.4 and blocks it from all other ports (0-52/tcp and 54-65535/tcp). I’m still not sure how to specify that I want to block all ports, or block both TCP and UDP traffic using this config option.

IPBan has a lot of other configuration options that I’m excited to test. For me, this tool fills a major gap for Windows servers!

Modifying a packet capture with Scapy

My motivation was to start from a known good packet capture, for example, a DNS request and reply, and modify that request to create something interesting: an example to examine in Wireshark, or positive and negative test cases for an IDS software (Snort, Suricata).

I haven’t done much with Scapy before, but it seemed like the right tool for the task. My planned steps were as follows:

  1. Take pcap (packet capture)
  2. Import pcap via scapy
  3. Modify pcap
  4. Export pcap
  5. View pcap in Wireshark

Continue reading Modifying a packet capture with Scapy

Identifying DGA domains using Scrabble scores: a naive approach

I had the idea of applying Scrabble scores to DGA domains over the summer of 2018, but the idea was rekindled when I saw Marcus Ranum‘s keynote at BroCon 2018. He talked about the advantages of scoring systems: they are fast, they are simple, and they can be surprisingly effective.

Domain Generating Algorithms (DGAs)

Malware uses DGAs to generate hundreds or thousands of new domain names daily. The malware then attempts to contact some or all of the domains. If a successful attempt is made to a control server, the malware will receive new instructions for malicious activity. The people and systems managing the malware need only register one new domain a day, but a defender would have to anticipate and/or discover thousands a day. To read more about DGAs, I recommend these articles from Akamai:

Scrabble Scores and DGAs

I’ve noticed that some, not all Domain Generating Algorithms produce unreadable domains like:

rjklaflzzdglveziblyvvcyk.com

Continue reading Identifying DGA domains using Scrabble scores: a naive approach

SELinux, audit2why, audit2allow, and policy files

I’m no expert on SELinux, but I cringe whenever I read an online tutorial that includes the step Disable SELinux.

I ran into such a problem recently when I was installing Icinga. The service failed to start because of permissions issues creating the process ID (PID) file. One site suggested disabling SELinux, but I thought it was time to learn to update SELinux’s Type Enforcement (TE) policies instead.

First, I needed the audit2why tool, to explain what was being blocked and why:

# yum -q provides audit2why
policycoreutils-python-2.5-17.1.el7.x86_64 : SELinux policy core python
                                           : utilities
Repo        : base
Matched from:
Filename    : /usr/bin/audit2allow

I installed the policycoreutils-python package (and dependencies):

# yum install policycoreutils-python

I then ran audit2why against the audit log:

# audit2why -i /var/log/audit/audit.log
type=AVC msg=audit(1510711476.690:132): avc:  denied  { chown } for  pid=2459 comm="icinga" capability=0  scontext=system_u:system_r:nagios_t:s0 tcontext=system_u:system_r:nagios_t:s0 tclass=capability

        Was caused by:
                Missing type enforcement (TE) allow rule.

                You can use audit2allow to generate a loadable module to allow this access.

type=AVC msg=audit(1510711476.724:134): avc:  denied  { read write } for  pid=2465 comm="icinga" name="icinga.pid" dev="tmpfs" ino=19128 scontext=system_u:system_r:nagios_t:s0 tcontext=system_u:object_r:initrc_var_run_t:s0 tclass=file

        Was caused by:
                Missing type enforcement (TE) allow rule.

                You can use audit2allow to generate a loadable module to allow this access.

That’s still a little opaque. It’s not entirely clear to me why chown was blocked, for example. Look at the following specifics:

scontext=system_u:system_r:nagios_t:s0
tcontext=system_u:system_r:nagios_t:s0

To help decode that:

  • scontext = Source Context
  • tcontext = Target Context
  • _u:_r:_t:s# = user:role:type:security level

The source and target contexts are identical, and so it seems to me that the command should be allowed. But let’s try audit2allow and see what that tells us:

# audit2allow -i /var/log/audit/audit.log


#============= nagios_t ==============
allow nagios_t initrc_var_run_t:file { lock open read write };
allow nagios_t self:capability chown;

It is unclear to me how broad the first rule is: does it allow the nagios type (nagios_t) access to all initrc_var_run_t files? If so, that’s probably too broad. As the man page warns:

Care must be exercised while acting on the output of  this  utility  to
ensure  that  the  operations  being  permitted  do not pose a security
threat. Often it is better to define new domains and/or types, or  make
other structural changes to narrowly allow an optimal set of operations
to succeed, as opposed to  blindly  implementing  the  sometimes  broad
changes  recommended  by this utility.

That’s fairly terrifying. Although if the alternative is disabling SELinux completely, an overly broad SELinux policy is not the worst thing in the world.

So audit2allow provided a couple rules. Now what? Fortunately the audit2why and audit2allow man pages both include details on how to incorporate the rules into your SELinux policy. First, generate a new type enforcement policy:

# audit2allow -i /var/log/audit/audit.log --module local > local.te

This includes some extra information in addition to the default output:

# cat local.te

module local 1.0;

require {
        type nagios_t;
        type initrc_var_run_t;
        class capability chown;
        class file { lock open read write };
}

#============= nagios_t ==============
allow nagios_t initrc_var_run_t:file { lock open read write };
allow nagios_t self:capability chown;

Next the man page says:

# SELinux provides a policy devel environment under
# /usr/share/selinux/devel including all of the shipped
# interface files.
# You can create a te file and compile it by executing

$ make -f /usr/share/selinux/devel/Makefile local.pp

However, my system had no /usr/share/selinux/devel directory:

# ls /usr/share/selinux/
packages  targeted

I needed to install the policycoreutils-devel package (and dependencies):

# yum install policycoreutils-devel

Now compile the policy file to a binary:

# make -f /usr/share/selinux/devel/Makefile local.pp
Compiling targeted local module
/usr/bin/checkmodule:  loading policy configuration from tmp/local.tmp
/usr/bin/checkmodule:  policy configuration loaded
/usr/bin/checkmodule:  writing binary representation (version 17) to tmp/local.mod
Creating targeted local.pp policy package
rm tmp/local.mod.fc tmp/local.mod

Now install it using the semodule command:

# semodule -i local.pp

Did that solve the problem?

# systemctl start icinga
# systemctl status icinga
● icinga.service - LSB: start and stop Icinga monitoring daemon
   Loaded: loaded (/etc/rc.d/init.d/icinga; bad; vendor preset: disabled)
   Active: active (running) since Tue 2017-11-14 22:35:23 EST; 6s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 2661 ExecStop=/etc/rc.d/init.d/icinga stop (code=exited, status=0/SUCCESS)
  Process: 3838 ExecStart=/etc/rc.d/init.d/icinga start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/icinga.service
           └─3850 /usr/bin/icinga -d /etc/icinga/icinga.cfg

Nov 14 22:35:23 localhost.localdomain systemd[1]: Starting LSB: start and sto...
Nov 14 22:35:23 localhost.localdomain icinga[3838]: Running configuration che...
Nov 14 22:35:23 localhost.localdomain icinga[3838]: Icinga with PID  not runn...
Nov 14 22:35:23 localhost.localdomain icinga[3838]: Starting icinga: Starting...
Nov 14 22:35:23 localhost.localdomain systemd[1]: Started LSB: start and stop...
Nov 14 22:35:23 localhost.localdomain icinga[3850]: Finished daemonizing... (...
Nov 14 22:35:23 localhost.localdomain icinga[3850]: Event loop started...
Hint: Some lines were ellipsized, use -l to show in full.

It worked! The permissions issues were resolved without resorting to disabling SELinux.

There is still more I need to understand about SELinux, but it’s a start.

Additional reading:
CentOS: SELinux Policy Overview

Using blocklist.de with fail2ban

Anyone who runs a server with open ports knows that systems with questionable intent will come knocking. Sometimes thousands of them. fail2ban is software that that checks your server logs and detects multiple failures, for example 5 failed SSH logins in a row, and bans the source IP address a period of time, e.g. for an hour. This helps prevent password-guessing and brute force attacks. It might be useful to share information about those questionable IP addresses with others so that we can block them proactively.

One such list of IP addresses that I found is blocklist.de. Since I am primarily concerned with systems that are trying to SSH into my system, I looked specifically at their SSH blocklist:
All IP addresses which have been reported within the last 48 hours as having run attacks on the service SSH.

Implementation details: Continue reading Using blocklist.de with fail2ban

Using fail2ban with iptables instead of firewalld

In the previous post I wrote about the minor configuration changes needed to get fail2ban to actually do something.

I have been working primarily with CentOS 7 and have been using iptables instead of firewalld. Normally, fail2ban works with iptables by default. However, installing fail2ban on CentOS 7 also installs fail2ban-firewalld — which changes that default. Even with a properly configured fail2ban jail, you will not see the expected results. fail2ban will log events as expected, but no traffic will actually be banned.

The fail2ban-firewalld package places a file in /etc/fail2ban/jail.d/00-firewalld.conf. It overrides the default banaction (iptables) and sets it to firewallcmd-ipset.

The top of the 00-firewalld.conf file says:

You can remove this package (along with the empty fail2ban meta-package) if you do not use firewalld

When I tried removing fail2ban-firewalld, it removed fail2ban as a dependency. I have a feeling the referenced fail2ban meta-package may have something to so with that.

I have not yet investigated the meta-package and de-coupling fail2ban-firewalld from fail2ban (see Update below). My solution, for now, has been to move 00-firewalld.conf and restart fail2ban:

$ sudo mv /etc/fail2ban/jail.d/00-firewalld.conf /etc/fail2ban/jail.d/00-firewalld.disabled
$ sudo systemctl restart fail2ban

The default banaction defined in jail.conf is no longer overridden and performs as expected:
banaction = iptables-multiport

Update
According to Fail2ban with FirewallD, The fail2ban package itself is a meta-package that contains several other packages, including fail2ban-firewalld and fail2ban-server. Removing the meta-package will not remove fail2ban-server.

If you’ve already moved 00-firewalld.conf to 00-firewalld.disabled, you’ll get a warning:
warning: file /etc/fail2ban/jail.d/00-firewalld.conf: remove failed: No such file or directory

You can ignore the warning, or remove 00-firewalld.disabled.

fail2ban fails to ban SSH login failures

fail2ban is one of those magical programs that, in my experience, just works. I’ve inherited many systems with a working fail2ban configuration, and therefore I didn’t know much about configuring it or troubleshooting it.

Summary: by default, fail2ban on CentOS 7 does absolutely nothing!

One of the things that it is reported (falsely!) to do out-of-the-box is to block repeated SSH login failures. According to Protecting SSH with Fail2ban:

Fail2ban should now protect SSH out of the box. If Fail2ban notices six failed login attempts in the last ten minutes, then it blocks that IP for ten minutes.

I wanted to test this, so I set up 2 virtual machines, a victim and an attacker.

On the victim VM:
[ariel]# sudo yum install epel-release
[ariel]# sudo yum install fail2ban
[ariel]# sudo systemctl start fail2ban
[ariel]# sudo tail -f /var/log/fail2ban

On the attacker VM:
[caliban]# sudo yum install epel-release
[caliban]# sudo yum install sshpass
[caliban]# for i in `seq 1 100`; do sshpass -p 'TopSecret!' admin@ariel; done

And then I waited. And waited. And waited.

I confirmed that the defaults described matched what was in my /etc/fail2ban/jails.conf (excerpted):
bantime = 600
findtime = 600
maxretry = 5

In my test, I definitely exceeded that: about 30 failed attempts in 5 minutes. The failures appear in /var/log/secure, but nothing appears in /var/log/fail2ban.log!

From How To Protect SSH With Fail2Ban on CentOS 7 I found the fail2ban-client status command:

[ariel]# fail2ban-client status
Status
|- Number of jail: 0
`- Jail list:

Zero jails! That’s definitely a problem.

As mentioned in the above, I created a file, /etc/fail2ban/jail.local containing the following:
[sshd]
enabled = true

New results:
[ariel]# systemctl restart fail2ban
[ariel]# fail2ban-client status
Status
|- Number of jail: 1
`- Jail list: sshd

That looks better! /var/log/fail2ban.log now has new entries, and the attacker IP address has been banned! Just to confirm I tried to SSH to the machine from the attacker:

[caliban]# ssh admin@ariel
ssh_exchange_identification: Connection closed by remote host

Great! Exactly what I expected to happen.

When I look at the /etc/fail2ban/jails.conf, I do not see enabled = true under the [sshd] section. In fact, part of that file explains that all jails are disabled by default:

# "enabled" enables the jails.
# By default all jails are disabled, and it should stay this way.
# Enable only relevant to your setup jails in your .local or jail.d/*.conf
#
# true: jail will be enabled and log files will get monitored for changes
# false: jail is not enabled
enabled = false

On CentOS 7, fail2ban is configured to work with firewalld. My next post describes using fail2ban with iptables on CentOS 7.

Let’s Encrypt: certbot error “No vhost exists with servername or alias of”

It’s about time–or rather, years past time–I enabled HTTPS for this site. I decided to try Let’s Encrypt. It wasn’t as turnkey as I expected, so I’ve included some notes here in case anyone else has similar issues.

The Let’s Encrypt site suggested installing Certbot and included specific instructions for using Certbot with Apache on CentOS 7. It suggested that a single command might do the trick:

$ sudo certbot --apache

Unfortunately, I received a couple error messages and it was ultimately able to create the certificate for me, but unable to update my Apache configuration. An excerpt of the output of the certbot command is below:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
No names were found in your configuration files. Please enter in your domain
name(s) (comma and/or space separated) (Enter 'c' to cancel):osric.com,www.osric.com
...
No vhost exists with servername or alias of: osric.com (or it's in a file with multiple vhosts, which Certbot can't parse yet). No vhost was selected. Please specify ServerName or ServerAlias in the Apache config, or split vhosts into separate files.
Falling back to default vhost *:443...
No vhost exists with servername or alias of: www.osric.com (or it's in a file with multiple vhosts, which Certbot can't parse yet). No vhost was selected. Please specify ServerName or ServerAlias in the Apache config, or split vhosts into separate files.
Falling back to default vhost *:443...
...
No vhost selected

IMPORTANT NOTES:
- Unable to install the certificate
...

I’m guessing it’s because my Apache virtual host configuration is in /etc/httpd/conf/vhosts/chris/osric.com instead of the expected location.

I looked at the certbot documentation hoping to find a way I could pass the certbot command the path to my virtual host configuration file. I did not find an option to do that. The logs at /var/log/letsencrypt/letsencrypt.log are fairly verbose, but it still does not indicate what files or directories it looked at to attempt to find my Apache configuration.

I noted that /etc/letsencrypt/options-ssl-apache.conf contains Apache directives. I thought maybe I could just include it in my config file using Apache’s Include directive, e.g.:

Include /etc/letsencrypt/options-ssl-apache.conf

I restarted Apache using systemctl (I know, I should be using apachectl restart instead):

$ sudo systemctl restart httpd
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.

Two problems there. One, options-ssl-apache.conf appears to be a generic file with no data specific to the host or cert. Additionally, I had just added it to a VirtualHost directive listening on port 80.

I duplicated the VirtualHost directive in my config file at /etc/httpd/conf/vhosts/chris/osric.com and made a few modifications and additions:

<IfModule mod_ssl.c>
<VirtualHost 216.154.220.53:443>
...all the directives from the port 80 VirtualHost...
SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/osric.com/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/osric.com/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/osric.com/chain.pem
</VirtualHost>
</IfModule>

I restarted Apache:

$ sudo apachectl restart

The server restarted, but still did not respond to HTTPS requests. It didn’t appear to be listening on 443:

$ curl https://www.osric.com
curl: (7) Failed connect to www.osric.com:443; Connection refused

As a sanity check, I confirmed that mod_ssl was indeed installed:

$ yum list mod_ssl
Installed Packages
mod_ssl.x86_64 1:2.4.6-45.el7.centos @base

And I checked to confirm that Apache was loading mod_ssl:

$ cat /etc/httpd/conf.modules.d/00-ssl.conf
LoadModule ssl_module modules/mod_ssl.so

I looked at some other Apache configurations where I knew SSL was working and I noted the Listen directive:

Listen 443

I added that line to the top of my configuration file at /etc/httpd/conf/vhosts/chris/osric.com, above the VirtualHost directive. I restarted Apache and it worked!

Browser metadata phishing?

I was checking my Google Analytics stats and noticed a strange entry in the Languages section of the demographics. Ranking fifth, after en-us, en-gb, en-ca, and en-au was the following:

Secret.ɢoogle.com You are invited! Enter only with this ticket URL. Copy it. Vote for Trump!

Do not visit that URL, by the way. You can see that the first “G” in “Google” is an unusual character — it’s the symbol for a voiced uvular stop.

I usually use urlQuery to check out potentially malicious sites, but it didn’t like this URL. I used vURL Online instead, which reported it was malicious:

This domain is listed in the Malware Domain List. Website’s [sic] in this database should be viewed with extreme caution.

These 1500 or so sessions on my site are presumably from some hijacked browser or malicious plug-in/extension, and the end-user has no idea they are sending this bizarre language string in the HTTP headers.

Why put a malicious URL there at all? Did the creator hope that those of us perusing our web stats would be intrigued enough to fall for this trap? Even as I ask that question, I know that some percentage of users must have done just that. I assume they are now broadcasting their language as the same unusual string.

As a site owner, is there anything I should do? I could detect this string and notify the user. E.g. use an Apache re-write rule to redirect the user to a page telling them their browser is infected? This is only a partially rhetorical question. If you have suggestions, let me know.