fail2ban fails to ban SSH login failures

fail2ban is one of those magical programs that, in my experience, just works. I’ve inherited many systems with a working fail2ban configuration, and therefore I didn’t know much about configuring it or troubleshooting it.

Summary: by default, fail2ban on CentOS 7 does absolutely nothing!

One of the things that it is reported (falsely!) to do out-of-the-box is to block repeated SSH login failures. According to Protecting SSH with Fail2ban:

Fail2ban should now protect SSH out of the box. If Fail2ban notices six failed login attempts in the last ten minutes, then it blocks that IP for ten minutes.

I wanted to test this, so I set up 2 virtual machines, a victim and an attacker.

On the victim VM:
[ariel]# sudo yum install epel-release
[ariel]# sudo yum install fail2ban
[ariel]# sudo systemctl start fail2ban
[ariel]# sudo tail -f /var/log/fail2ban

On the attacker VM:
[caliban]# sudo yum install epel-release
[caliban]# sudo yum install sshpass
[caliban]# for i in `seq 1 100`; do sshpass -p 'TopSecret!' admin@ariel; done

And then I waited. And waited. And waited.

I confirmed that the defaults described matched what was in my /etc/fail2ban/jails.conf (excerpted):
bantime = 600
findtime = 600
maxretry = 5

In my test, I definitely exceeded that: about 30 failed attempts in 5 minutes. The failures appear in /var/log/secure, but nothing appears in /var/log/fail2ban.log!

From How To Protect SSH With Fail2Ban on CentOS 7 I found the fail2ban-client status command:

[ariel]# fail2ban-client status
Status
|- Number of jail: 0
`- Jail list:

Zero jails! That’s definitely a problem.

As mentioned in the above, I created a file, /etc/fail2ban/jail.local containing the following:
[sshd]
enabled = true

New results:
[ariel]# systemctl restart fail2ban
[ariel]# fail2ban-client status
Status
|- Number of jail: 1
`- Jail list: sshd

That looks better! /var/log/fail2ban.log now has new entries, and the attacker IP address has been banned! Just to confirm I tried to SSH to the machine from the attacker:

[caliban]# ssh admin@ariel
ssh_exchange_identification: Connection closed by remote host

Great! Exactly what I expected to happen.

When I look at the /etc/fail2ban/jails.conf, I do not see enabled = true under the [sshd] section. In fact, part of that file explains that all jails are disabled by default:

# "enabled" enables the jails.
# By default all jails are disabled, and it should stay this way.
# Enable only relevant to your setup jails in your .local or jail.d/*.conf
#
# true: jail will be enabled and log files will get monitored for changes
# false: jail is not enabled
enabled = false

On CentOS 7, fail2ban is configured to work with firewalld. My next post describes using fail2ban with iptables on CentOS 7.

Browser metadata phishing?

I was checking my Google Analytics stats and noticed a strange entry in the Languages section of the demographics. Ranking fifth, after en-us, en-gb, en-ca, and en-au was the following:

Secret.ɢoogle.com You are invited! Enter only with this ticket URL. Copy it. Vote for Trump!

Do not visit that URL, by the way. You can see that the first “G” in “Google” is an unusual character — it’s the symbol for a voiced uvular stop.

I usually use urlQuery to check out potentially malicious sites, but it didn’t like this URL. I used vURL Online instead, which reported it was malicious:

This domain is listed in the Malware Domain List. Website’s [sic] in this database should be viewed with extreme caution.

These 1500 or so sessions on my site are presumably from some hijacked browser or malicious plug-in/extension, and the end-user has no idea they are sending this bizarre language string in the HTTP headers.

Why put a malicious URL there at all? Did the creator hope that those of us perusing our web stats would be intrigued enough to fall for this trap? Even as I ask that question, I know that some percentage of users must have done just that. I assume they are now broadcasting their language as the same unusual string.

As a site owner, is there anything I should do? I could detect this string and notify the user. E.g. use an Apache re-write rule to redirect the user to a page telling them their browser is infected? This is only a partially rhetorical question. If you have suggestions, let me know.

Social Engineering through Surveys

I received an invitation to a survey today. I was selected as an alumnus of the University of Michigan, an enormous university. The e-mail implies that the survey is possibly on behalf of the university. It includes the well-recognized “Block M” logo.

However:

  • The “From” address is alumnisurvey@lrwonlinesurvey.com.
  • Links to unsubscribe go to click.skem1.com.
  • The survey itself is at survey.bz.

It all looks pretty fishy/phishy.

Nowhere are there any links to umich.edu.

Also, I happen to know that the University of Michigan tends to use Qualtrics for surveys. Why wouldn’t the university send out a Qualtrics survey from a umich.edu e-mail address with umich.edu unsubscribe links instead of a survey.bz survey from a lrwonlinesurvey.com address with click.skem1.com unsubscribe links?

The survey is likely legitimate. The alumni department probably contracted with a research firm, that research firm probably uses a third-party survey software, and they probably use a different third-party service to handle mailing lists.

But I will not be filling out such a survey. You shouldn’t either. And, if you’re in the business of creating surveys or hiring companies to create surveys, you should think about these factors. Why create something that looks this suspicious?

I’ve always said that survey results automatically exclude those who don’t have time to waste on surveys (this one suggested it would take 18 minutes to complete!), but now it seems they also exclude anyone with a mind for security and privacy.

ColdFusion session fixation and jsessionid

A web application is vulnerable to a session fixation attack if an unauthenticated user’s session ID does not change after authentication. A malicious user could start an unauthenticated session and give the associated session ID to the victim. Once the victim authenticates, the malicious user now shares that authenticated session.

ColdFusion introduced SessionRotate in CF10 to mitigate such attacks. However, it only applies to CFID and CFTOKEN (ColdFusion sessions). If you are using J2EE sessions, there is no SessionRotate. This is unfortunate, as the ColdFusion documentation has stated for years that J2EE sessions have advantages over ColdFusion sessions (see Configuring and using session variables), the foremost being that J2EE sessions are not persistent. It’s also unfortunate, because this is an important security-related tag that only works under certain server configurations–and fails silently under others!

As a side note, J2EE has been called JEE (Java Enterprise Edition) since Java 1.4. The ColdFusion administrator continues to refer to it as J2EE, but you are likely to see references to JEE in reference material online.

Session rotation basically consists of:

  • Deleting the user’s current session
  • Creating a new session containing the same data, but with a different ID

Jason Dean at 12robots.com wrote about mitigating session fixation on ColdFusion in Session token rotation REVISITED and other articles. His posts were helpful in explaining the issue and offering a solution for rotating ColdFusion sessions on pre-CF10 systems. However, he posted no solution for JEE sessions.

But according to the CF documentation on Ending a session: You cannot destroy the session and create a session on the same request, as creating a new session involves sending session cookies back.

That’s problematic. Once a user authenticates, we can destroy the user’s session. But how do we create the new, authenticated session?

An answer posted in response to ColdFusion Session Fixation on StackOverflow suggests the solution is to store necessary data in a secure way and pass the data, or a token associated with that data, on to a new HTTP request. I considered, for example, generating a UUID as a nonce and storing it in a database along with the username and a near-future timestamp. The user would then be directed to a URL with the nonce appended as a token, which could be validated against the database to authenticate the user.

Fortunately, a simpler solution exists. Pete Freitag posted SessionRotate solution for JEE Sessions in March, 2014. By accessing properties of the Java servlet session running under ColdFusion, his methods perform the following steps:

  • Copy the session scope into a temporary variable
  • Remove the ID and URL tokens from the temporary variable
  • Invalidate the existing session
  • Create a new session
  • Copy the attributes of the temporary variable into the new session

The final step is somewhat more complicated, as the new session is apparently not immediately available, and copying the attributes happens in the Application.cfc’s OnSessionStart. Still, it can happen in a single request! The JEE documentation on Creating and Managing User Sessions has more information about the underlying JEE sessions.

When I implemented Pete Freitag’s solution, I made the following modifications:

  • I created a jeeSession.cfc containing:
    1. rotate (originally named jeeSessionRotate)
    2. copy (containing most of onSessionStart)
  • I created jeeSecureApp.cfc containing:
    1. onSessionStart
    2. an instance of jeeSession.cfc

I can modify the cfcomponent tag in the Application.cfc of any app to include:
extends="jeeSecureApp"

Then jeeSession.rotate() is available.

Caveats:

  • If your application already extends a CFC, you’d have to adjust this approach.
  • If your application already defines an onSessionStart, you’d want to add a call to super.onSessionStart

SQL Injection Goes Mainstream

Note to the web development world: even a mainstream media source like Time Magazine knows about SQL injection. Don’t you think it’s time you protected your web applications against it?

Last week’s issue of Time featured an article that focused on LulzSec‘s activities. In Hack Attack, author Bill Saporito went a step beyond most journalists covering web security by mentioning an actual technique: SQL injection.

SQL injection is a subclass of injection attacks, wherein a malicious user manipulates input so as to insert (or inject) a tag-along command into the application code. It’s #1 on OWASP’s Top Ten Vulnerabilities for 2010, in part because such vulnerabilities are:

  • Common
  • Easy to exploit
  • Have a huge payoff (i.e. devastating impact)

Because it is so common and easy-to-exploit, there are a lot of automated tools that malicious users (often by way of compromised machines) use to scan sites and test them for vulnerabilities. If a vulnerability is found, the application may be targeted for further attacks. Basically, attackers are on the lookout for low-hanging fruit in the same way that thieves look for valuables sitting in plain sight in a parked car. Don’t take the car analogy too far, though: if someone breaks into your car, there will be broken glass and your iPod will be gone. If someone exploits a SQL injection vulnerability on your site, they may have all your user data (and more), with hardly a trace: entries in your access logs and error logs, which are too-often completely ignored.

As the joke goes, you don’t have to outrun the bear to avoid being mauled and eaten–you just have to outrun the other guy. One of the best ways to make sure your web application is not targeted for further attacks is to make sure the relatively simple SQL injection scanners don’t find any vulnerabilities.

SQL injection is fairly simple to defend against using parameterized input, and your development language of choice should have documentation on how to do this. OWASP also offers a SQL Injection Prevention Cheat Sheet. There are also automated tools to you can use to check your code for SQL injection flaws (such as QueryParam Scanner for ColdFusion), or test your site for vulnerabilities–also known as penetration testing or pen testing–such as the (currently out-of-date) SQL Inject Me add-on for Firefox.

Checking your web applications for SQL injection vulnerabilities is the first thing you should do, but it is hardly the last. Although fending off automated SQL injection attempts is definitely a good thing, there are many other categories of vulnerabilities, and a determined attacker will find them. Stay informed, and make sure you know what attackers are up to before you read about it in Time.

Why I’m Not Friends with My Bank on Facebook

I received a request today from my financial institution asking me to follow them on Facebook, Twitter, Flickr, and YouTube.

Aside from the fact that I doubt that their updates on these various services will enrich my life, there is another very good reason not to follow them:

Security.

It’s easy to trace your connections online. Most of this information, for most users, is public. If you follow Bank A, it stands to reason that you have an account at Bank A–something a malicious person would not have known before. Even if your online persona isn’t directly connected to your name, you might be surprised at how easy it is to connect the two with a Google search.

(That last item says a lot, I think.)

Any bank that suggests you follow them on social media must be pretty confident of their security! Or, more likely, their marketing teams and their security teams don’t talk to each other.

You wouldn’t stand on a street-corner handing out cards that say, “My name is Bob Billiards and I have an account at Bank A” would you? Then don’t follow your bank on a social media site.

WordPress Security Tips

I attended the WordCamp Birmingham conference this past weekend to find out more about all things WordPress. WordPress is an open source blog engine/lightweight Content Management System (CMS) that has a huge community of users and developers, and an enormous repository of plugins to extend its functionality.

One of the presentations I attended, Mitch Canter‘s session on WordPress Security, had 6 good tips for making your WordPress-based site more secure:
Continue reading WordPress Security Tips

Validating the Referer: Not as Useless as I Thought?

I used to validate the HTTP referer header to verify that users were accessing certain pages from certain other pages. For example, users accessing sampleapp/edit.cfm should be getting there from sampleapp/index.cfm. Anyone accessing sampleapp/edit.cfm without coming from sampleapp/index.cfm was probably monkeying around and should be send back to the index page, or possibly even logged out.

However, it is fairly trivial to modify your referer header, so anyone who wants to monkey around with sampleapp/edit.cfm can make it look like they are coming from sampleapp/index.cfm. (If you’re interested in modifying your HTTP headers, I suggest checking out the Tamper Data Firefox plugin.) The check provides absolutely no assurance that the user is really coming from the page. Therefore, I decided the check was useless.

I’ve been attending a weekly web application security study group with some of my colleagues for the past several weeks, where we’ve been reading and discussing The Web Application Hacker’s Handbook. The past couple sessions have been about cross-site scripting (XSS). Justin Klein Keane brought up a good point at today’s session: checking the referer may not keep a malicious user from altering his or her referer string, but could help identify victims of XSS attacks who were possibly directed to submit malicious data from a third-party site.

Checking the referer isn’t a sufficient protection against malicious users, by any means, but it could still be helpful. What do you think?

Encrypted versus hashed passwords

I’m trying to decide whether it is better to store passwords in a database as key-encrypted strings, or as the result of a hash function (with salt).

Padlock

An encrypted string is secure as long as the key is secure, which it seems to me is both its strength and its Achilles’ heel. Since the application that accesses the database needs to use the key, that means that if both the database and the application server are compromised, the data is compromised.

It also means that if the application developers have access to the database, or if the DBAs have access to the application code, the data is available to those individuals. Even though those users are most likely trustworthy, it is perhaps an unnecessary risk–not to mention their workstations may be compromised. On top of that, you need to worry about key escrow–if you lose the key, you no longer have access to your own data.

A hashed value is secure even if the database and the application are compromised, as it is the result of a one-way function. The value is also inaccessible to either the application developers or the DBAs, which provides an additional layer of security. On the other hand, since it uses a well-known hash function, such as MD5, creating a dictionary file of hashed values is trivial, and apparently an even more effective method to reveal hashed data is to use a rainbow table (the description of which goes a bit over my head right now).

That’s where the salt comes in: concatenate the original value along with extra data before creating the hash. I can see how this would foil a simple dictionary of hashed values. From what I’ve read, the salt value can be unique for each hashed value, and can be stored alongside the hashed value in the database. I’m not sure I entirely understand how that defends against rainbow tables, but it sounds good to me.

Using a hashed value, you can’t retrieve the original data–you can only match against it. This would not work well for data that you need to access again in its original form, e.g. phone numbers. This is a drawback in some cases, but probably not for passwords.

Right now I’m leaning towards salted hash over encryption, but maybe that’s because I’m hungry and it sounds more like breakfast. I’d love to know what other people think.