Block an IP address via iptables

I was monitoring the mail logs on a Postfix server and noted repeated failed connection attempts from the same IP address. The source was likely up to no good, and it was making it more difficult to monitor the logs for legitimate connections, so I decided to block it:

iptables -A INPUT -s 123.456.789.101 -j DROP

(IP address changed to protect…the innocent?)

However, the IP address was still making connections:
Dec 2 17:19:05 mercutio postfix/smtpd[15230]: connect from unknown[123.456.789.101]
Dec 2 17:19:06 mercutio postfix/smtpd[15230]: lost connection after AUTH from unknown[123.456.789.101]
Dec 2 17:19:06 mercutio postfix/smtpd[15230]: disconnect from unknown[123.456.789.101]

How is that possible? First I checked iptables to check my sanity and confirm that the rule had been added:

# iptables -L
...
DROP all -- 123.456.789.101 anywhere
...

OK, it’s there. That’s good!

The problem in this case was a different rule that had been added previously. Rules in iptables are processed in order, and no further rules are processed after a matching rule is found. Well above my newly-added rule was this rule:
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:smtp

That rule makes sense for a mail server, but I needed my rule to be inserted before it. I determined which rule it was in the INPUT chain like this:
iptables --line-numbers -L INPUT

It was the 5th rule, so I was able to insert the new rule just above it like this:
iptables -I INPUT 4 -s 123.456.789.101 -j DROP

After that, the offending IP address stopped creating entries in the mail.log.

However, my new rule would disappear after a system restart. Since I am using iptables-persistent, I saved the rules to the config file:
iptables-save > /etc/iptables/rules.v4

To confirm everything worked, I attempted to restart iptables:
# service iptables-persistent restart
Failed to restart iptables-persistent.service: Unit iptables-persistent.service

Apparently the service name changed to netfilter-persistent in Debian 8. The config files are still in the same location, but the service name has changed.

I restarted iptables:
# service netfilter-persistent restart

I checked the rules again and my new rule was there, above the rule allowing connections from any IP on port 25. However, I also noticed the following rule above either of those:
ACCEPT all -- anywhere anywhere

I freaked out. That rule indicates that all traffic from any source on any port should be accepted. That’s the worst firewall rule I’ve ever seen. It basically negates the entire concept of a firewall. It clearly should not be there!

However, using the verbose switch on iptables:
iptables -vL INPUT

I discovered that the rule only applied to the l0 interface (loopback). That’s a relief–that rule gets to stay.

Social Engineering through Surveys

I received an invitation to a survey today. I was selected as an alumnus of the University of Michigan, an enormous university. The e-mail implies that the survey is possibly on behalf of the university. It includes the well-recognized “Block M” logo.

However:

  • The “From” address is alumnisurvey@lrwonlinesurvey.com.
  • Links to unsubscribe go to click.skem1.com.
  • The survey itself is at survey.bz.

It all looks pretty fishy/phishy.

Nowhere are there any links to umich.edu.

Also, I happen to know that the University of Michigan tends to use Qualtrics for surveys. Why wouldn’t the university send out a Qualtrics survey from a umich.edu e-mail address with umich.edu unsubscribe links instead of a survey.bz survey from a lrwonlinesurvey.com address with click.skem1.com unsubscribe links?

The survey is likely legitimate. The alumni department probably contracted with a research firm, that research firm probably uses a third-party survey software, and they probably use a different third-party service to handle mailing lists.

But I will not be filling out such a survey. You shouldn’t either. And, if you’re in the business of creating surveys or hiring companies to create surveys, you should think about these factors. Why create something that looks this suspicious?

I’ve always said that survey results automatically exclude those who don’t have time to waste on surveys (this one suggested it would take 18 minutes to complete!), but now it seems they also exclude anyone with a mind for security and privacy.

iptables and deleting/replacing entries

Whenever I have to reboot my modem [sic] at home, I typically get a new IP address from my ISP.

When that happens, I need to update iptables to allow my new address to connect to the SSH port (port 22) of my jump box (which, fortunately, I have access to from another IP address):

iptables -A INPUT -p tcp -m state --state NEW -s [new IP address] --dport 22 -j ACCEPT

But I don’t want to leave the old entry. How to get rid of it?

The delete (-D) and replace (-R) options require a line number from the chain (e.g. the INPUT chain). To find the line numbers:

iptables -L INPUT --line-numbers

To delete the existing rule and add the new rule:

iptables -D INPUT [line number]
iptables -A INPUT -p tcp -m state --state NEW --dport 22 -s [new IP address] -j ACCEPT

To replace the existing entry:

iptables -R INPUT [line number] -p tcp -m state --state NEW --dport 22 -s [new IP address] -j ACCEPT

Save the updates so they are persistent:

iptables-save > /etc/iptables/rules.v4

(That’s the location for Debian and Ubuntu. This may be different for your distribution.)

UPDATE rows with values from a table JOIN in Oracle

Example use case: I have a database that contains a table of contacts (contact) and table of e-mail addresses (email), joined on contact.id = email.contact_id. I just found out that Example Conglomerate acquired Osric Publishing’s Oracle consulting business, and so I need to update my contacts database so that all of the Oracle consultants who had @osric.com e-mail addresses now have @example.com e-mail addresses.

How can I change just the affected addresses in the contact database, assuming the username portion of their e-mail addresses remains the same?
Continue reading UPDATE rows with values from a table JOIN in Oracle

3 ways to iterate over lines of a file in Linux

Frequently I need to run a process for each item in a list, stored in a text file one item per line: usernames, filenames, e-mail addresses, etc. Obviously there are more than 3 ways to do this, but here are 3 I have found useful:

Bash
sh prog1.sh list.txt

Source: prog1.sh

while read line
do
    echo $line
done < $1

4 lines. Not bad.

Perl
perl prog2.pl list.txt

Source: prog2.pl

while(<>) {
    print `echo $_`;
}

3 lines. Pretty good.

Perl -n
perl -n prog3.pl list.txt

Source: prog3.pl

print `echo $_`;

1 line! The -n switch basically wraps your Perl code in a loop that processes each line of the input file. I just discovered this while flipping through my 17-year-old copy of Programming Perl (link is to a newer edition).

I really like this method because you can write a script that processes a single input that could easily be reused by another script, but can also easily be used to process an entire list by adding just the -n switch. (There’s also a similar -p switch that does the same thing, but additionally prints out each line.)

I should note that in the examples above, I am using echo as a substitute for any command external to the script itself. In the Perl examples, there would be no need to call echo to merely print the contents of the line, but it’s a convenient stand-in for a generic command.

As suggested by a comment on a previous post, I have made these examples available in a git repository: iterate over lines.

Mail Users in Office 365 don’t have SMTP access

On-premises mail users (at least in Exchange 2010) had access to send mail as their organizational address through the on-premises SMTP server. However, mail users in Exchange Online cannot send mail as their organizational address using smtp.office365.com.

So what can we do?

In order to use SMTP, users need full mailboxes. But these users should not actually have mailbox access. As a test, created a mailbox and I disabled all email apps for the mailbox in Exchange Online:

An Office 365 user's email app settings
An Office 365 user’s email app settings

The test user was no longer able to log in to Outlook on the Web (also known as OWA). SMTP still worked. Email forwarding still worked (although the user would not be able to set the forwarding address themselves).

Creating a user mailbox requires a user license, whereas mail users do not require a license. If you have a lot of on-premises mail users that now need full mailboxes, this could be problematic.

Summary:
In Exchange Online, a UserMailbox with all email apps disabled is equivalent to an on-premises Exchange MailUser, except that the former requires a license.

Removing exceptions from a list using Bash (with sed and awk)

  • I have a CSV file, a list of 1000+ users and user properties.
  • I have a list of exceptions (users to be excluded from processing), one user per line, about 50 total.

How can I remove the exceptions from the list?

# make a copy of the original list
cp list-of-1000.csv list-of-1000-less-exceptions.csv
# loop through each line in exceptions.txt and remove matching lines from the copy
while read line; do sed -i "/${line}/d" list-of-1000-less-exceptions.csv; done < exceptions.txt

This is a little simplistic and could be a problem if any usernames are subsets of other usernames. (For example, if user ‘bob’ is on the list of exceptions, but the list of users also contains ‘bobb’, both would be deleted.)

In the particular instance I am dealing with, the username is conveniently the first field in the CSV file. This allows me to match the start of the line and the comma following the username:

while read line; do sed -i "/^${line},/d" list-of-1000-less-exceptions.csv; done < exceptions.txt

What if the username was the third field in the CSV instead of the first?

Use awk:
awk -F, -vOFS=, '{print $3,$0}' list-of-exceptions.csv > copy-of-list-of-exceptions.csv

  • -F, sets the field separator to a comma (defaults to whitespace)
  • -vOFS=, sets the Output Field Separator (OFS) to a comma (defaults to a space)
  • $3 prints the third field
  • $0 prints all the fields, with the specified field separator between them

while read line; do sed -i "/^${line},/d" copy-of-1000-less-exceptions.csv; done < exceptions.txt

Now there’s still an extra username in this file. Maybe that doesn’t matter, but maybe it does. There are several ways to remove it–here’s one:

awk -F, -vOFS=, '$1=""; print $0' copy-of-1000-less-exceptions.csv | sed 's/^,//' > list-of-1000-less-exceptions.csv

  • -F, sets the field separator to a comma (defaults to whitespace)
  • -vOFS=, sets the Output Field Separator (OFS) to a comma (defaults to a space)
  • $1="" sets the first field to an empty string
  • print $0 prints all the fields

The result of the awk command has an initial comma on each line. The first field is still there, it’s just set to an empty string. I used sed to remove it.

You could also use sed alone to remove the extra username field:
sed -i 's/^[^,]*,//' copy-of-1000-less-exceptions.csv

Middleman error – `block in replace_gem’: middleman-cli is not part of the bundle. Add it to Gemfile.

I ran into an error while setting up a Middleman site on my PC. I already had Ruby and RubyGems installed, so I followed the instructions to install Middleman and start a new Middleman site:

gem install middleman
middleman init project
cd project

The next step was to start the preview web server, which produced an error:

$ bundle exec middleman server
DL is deprecated, please use Fiddle
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/bundler-1.7.7/lib/bundler/r
ubygems_integration.rb:256:in `block in replace_gem': middleman-cli is not part
of the bundle. Add it to Gemfile. (Gem::LoadError)
from c:/RailsInstaller/Ruby2.1.0/bin/middleman:22:in `<main>'</main>

Continue reading Middleman error – `block in replace_gem’: middleman-cli is not part of the bundle. Add it to Gemfile.

Node.js – Error ‘console’ is undefined

I just installed Node.js on a new system and wanted to make sure it worked. I tested it with a simple hello world script.

Contents of the node.js file
console.log("Hello node");

Then I attempted to run it:
C:\>node node.js

Which produced the following error:
Script: C:\node.js
Line: 1
Char: 1
Error: 'console' is undefined
Code: 800A1391
Source: Microsoft JScript runtime error

Weird. And Microsoft JScript runtime? Very weird.

Turns out, Windows is trying to run the Javascript, node.js, not via Node.js but natively in Windows.

If I rename the file hellonode.js I can reproduce the same error:
C:\>hellonode

Or I can run the intended file via Node.js:
C:\>node hellonode.js
Hello node

I did not realize that the Windows command prompt could/would execute Javascript files natively.