FreeIPA: Could not chdir to home directory /home/bbilliards: no such file or directory

I recently installed a FreeIPA server and a FreeIPA client. I generated a Kerberos ticket for a test user, Bob Billiards, on the IPA server:

# kinit bbilliards
Password for bbilliards@IPA.OSRIC.NET:

Then I attempted to ssh into the IPA client as that user. The connection was successful, but it could not find the user’s home directory:

# ssh bbilliards@ariel.osric.net
bbilliards@ariel.osric.net's password:
Could not chdir to home directory /home/bbilliards: no such file or directory

The location of the home directory was set when I created the user, as can be seen here:

# ipa user-find bbilliards
--------------
1 user matched
--------------
  User login: bbilliards
  First name: Bob
  Last name: Billiards
  Home directory: /home/bbilliards
  Login shell: /bin/sh
  Principal name: bbilliards@IPA.OSRIC.NET
  Principal alias: bbilliards@IPA.OSRIC.NET
  Email address: bbilliards@ipa.osric.net
  UID: 1110200001
  GID: 1110200001
  SSH public key fingerprint: [redacted]
  Account disabled: False
----------------------------
Number of entries returned 1
----------------------------

Shouldn’t the system be able to create the home directory automatically? It turns out it can, if you specify the --mkhomedir switch when installing the IPA client:

# ipa-client-install --mkhomedir

Now when I ssh into the machine it creates a home directory:

# ssh bbilliards@ariel.osric.net
Creating home directory for bbilliards
-sh-4.2$ pwd
/home/bbilliards

You may prefer to mount a Network File System (NFS) directory as a home directory instead so that users have the same home directories across machines.

Error: Cannot contact any KDC for realm while getting initial credentials

I’ve been testing FreeIPA on a small network of CentOS 7 hosts (all virtual machines running in VirtualBox on a host-only network). After installing the IPA server on one host and creating the realm (IPA.OSRIC.NET), I installed the IPA client on one of the other hosts and tried running kinit:

# kinit admin
kinit: Cannot contact any KDC for realm 'IPA.OSRIC.NET' while getting initial credentials

Searching for that error brought me to Kinit won’t connect to a domain server. Although that did not describe the same issue, it did point me to the /etc/krb5.conf file. The realms section looked like it was missing something:

[realms]
  IPA.OSRIC.NET = {
    pkinit_anchors = FILE:/etc/ipa/ca.crt

  }

I added a kdc attribute:

[realms]
  IPA.OSRIC.NET = {
    kdc = prospero.osric.net:88
    pkinit_anchors = FILE:/etc/ipa/ca.crt
 
  }

No restart of any service was necessary. I ran kinit again and it worked:

# kinit admin
Password for admin@IPA.OSRIC.NET:

According to the krb5.conf documentation on realms:

kdc
The name or address of a host running a KDC for that realm. An optional port number, separated from the hostname by a colon, may be included.

I’m a Kerberos novice, but that seems like a necessary property. I’m not sure why the IPA client setup did not include it. I have a few more virtual machines to install the client on, so I’ll soon find if that behavior is consistent on subsequent installations.

check_http returns 403 Forbidden on fresh Nagios installation

I recently installed a Nagios server on a new CentOS 7 virtual machine (on Virtual Box).

One of the default checks included upon installation is a check on localhost to confirm that the HTTP server is responding. (First I had to install the check_http plugin, see previous post.) The Nagios web interface reports a warning for this check:

HTTP WARNING: HTTP/1.1 403 Forbidden - 5261 bytes in 0.001 second response time

This is unexpected, since I can request the same page in a browser, which returns the Apache Welcome page.

When I run the check manually I get the same result, as expected:

# /usr/lib64/nagios/plugins/check_http -H localhost
HTTP WARNING: HTTP/1.1 403 Forbidden - 5261 bytes in 0.001 second response time |time=0.000907s|;;;0.000000 size 5261B;;;0

I checked with curl:

# curl http://localhost

This returns the HTML source of the Apache Welcome page. It looks like it is working, right? But looking at the headers returned by the Apache server also shows 403 Forbidden:

# curl -I http://localhost
HTTP/1.1 403 Forbidden
...

The Apache Welcome page gives some hints about this behavior:

Are you the Administrator?

You should add your website content to the directory /var/www/html/.

To prevent this page from ever being used, follow the instructions in the file /etc/httpd/conf.d/welcome.conf.

The /etc/httpd/conf.d/welcome.conf file begins with the following comments and directive:

#
# This configuration file enables the default "Welcome" page if there
# is no default index page present for the root URL.  To disable the
# Welcome page, comment out all the lines below.
#
# NOTE: if this file is removed, it will be restored on upgrades.
#
<LocationMatch "^/+$">
    Options -Indexes
    ErrorDocument 403 /.noindex.html
</LocationMatch>

The Apache config is specifying that if there is no index page for the document root, return the Welcome page as an error document with a 403 HTTP status code.

Once I added a basic HTML file at /var/www/html/index.html, Nagios returned a success message:

HTTP OK: HTTP/1.1 200 OK - 549 bytes in 0.001 second response time

Missing Nagios plugins in CentOS 7

I set up a Nagios server on a CentOS 7 VM (Virtual Machine):

sudo yum install epel-release
sudo yum install nrpe
sudo yum install nagios

By default it sets up some basic checks for localhost. When I checked the Nagios site at http://127.0.0.1/nagios/, I found that even PING was critical:

(No output on stdout) stderr: execvp(/usr/lib64/nagios/plugins/check_ping, ...) failed. errno is 2: No such file or directory

I checked the contents of the plugins directory:

# ls /usr/lib64/nagios/plugins
eventhandlers negate urlize utils.sh

Sure enough, the usual suspects are not there. E.g.:

  • check_load
  • check_ping
  • check_disk
  • check_http
  • check_procs

Eventually I stumbled onto the following document, /usr/share/doc/nagios-plugins-2.0.3/README.Fedora:

Fedora users

Nagios plugins for Fedora have all been packaged separately. For
example, to isntall the check_http just install nagios-plugins-http.

All plugins are installed in the architecture dependent directory
/usr/lib{,64}/nagios/plugins/.

I installed some of the plugins following that convention:

sudo yum install nagios-plugins-load
sudo yum install nagios-plugins-ping
sudo yum install nagios-plugins-disk
sudo yum install nagios-plugins-http
sudo yum install nagios-plugins-procs

Now the the corresponding plugins exist in /usr/lib64/nagios/plugins, and Nagios reports OK for those checks on localhost.

Browser metadata phishing?

I was checking my Google Analytics stats and noticed a strange entry in the Languages section of the demographics. Ranking fifth, after en-us, en-gb, en-ca, and en-au was the following:

Secret.ɢoogle.com You are invited! Enter only with this ticket URL. Copy it. Vote for Trump!

Do not visit that URL, by the way. You can see that the first “G” in “Google” is an unusual character — it’s the symbol for a voiced uvular stop.

I usually use urlQuery to check out potentially malicious sites, but it didn’t like this URL. I used vURL Online instead, which reported it was malicious:

This domain is listed in the Malware Domain List. Website’s [sic] in this database should be viewed with extreme caution.

These 1500 or so sessions on my site are presumably from some hijacked browser or malicious plug-in/extension, and the end-user has no idea they are sending this bizarre language string in the HTTP headers.

Why put a malicious URL there at all? Did the creator hope that those of us perusing our web stats would be intrigued enough to fall for this trap? Even as I ask that question, I know that some percentage of users must have done just that. I assume they are now broadcasting their language as the same unusual string.

As a site owner, is there anything I should do? I could detect this string and notify the user. E.g. use an Apache re-write rule to redirect the user to a page telling them their browser is infected? This is only a partially rhetorical question. If you have suggestions, let me know.

Block an IP address via iptables

I was monitoring the mail logs on a Postfix server and noted repeated failed connection attempts from the same IP address. The source was likely up to no good, and it was making it more difficult to monitor the logs for legitimate connections, so I decided to block it:

iptables -A INPUT -s 123.456.789.101 -j DROP

(IP address changed to protect…the innocent?)

However, the IP address was still making connections:
Dec 2 17:19:05 mercutio postfix/smtpd[15230]: connect from unknown[123.456.789.101]
Dec 2 17:19:06 mercutio postfix/smtpd[15230]: lost connection after AUTH from unknown[123.456.789.101]
Dec 2 17:19:06 mercutio postfix/smtpd[15230]: disconnect from unknown[123.456.789.101]

How is that possible? First I checked iptables to check my sanity and confirm that the rule had been added:

# iptables -L
...
DROP all -- 123.456.789.101 anywhere
...

OK, it’s there. That’s good!

The problem in this case was a different rule that had been added previously. Rules in iptables are processed in order, and no further rules are processed after a matching rule is found. Well above my newly-added rule was this rule:
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:smtp

That rule makes sense for a mail server, but I needed my rule to be inserted before it. I determined which rule it was in the INPUT chain like this:
iptables --line-numbers -L INPUT

It was the 5th rule, so I was able to insert the new rule just above it like this:
iptables -I INPUT 4 -s 123.456.789.101 -j DROP

After that, the offending IP address stopped creating entries in the mail.log.

However, my new rule would disappear after a system restart. Since I am using iptables-persistent, I saved the rules to the config file:
iptables-save > /etc/iptables/rules.v4

To confirm everything worked, I attempted to restart iptables:
# service iptables-persistent restart
Failed to restart iptables-persistent.service: Unit iptables-persistent.service

Apparently the service name changed to netfilter-persistent in Debian 8. The config files are still in the same location, but the service name has changed.

I restarted iptables:
# service netfilter-persistent restart

I checked the rules again and my new rule was there, above the rule allowing connections from any IP on port 25. However, I also noticed the following rule above either of those:
ACCEPT all -- anywhere anywhere

I freaked out. That rule indicates that all traffic from any source on any port should be accepted. That’s the worst firewall rule I’ve ever seen. It basically negates the entire concept of a firewall. It clearly should not be there!

However, using the verbose switch on iptables:
iptables -vL INPUT

I discovered that the rule only applied to the lo interface (loopback). That’s a relief–that rule gets to stay.

Social Engineering through Surveys

I received an invitation to a survey today. I was selected as an alumnus of the University of Michigan, an enormous university. The e-mail implies that the survey is possibly on behalf of the university. It includes the well-recognized “Block M” logo.

However:

  • The “From” address is alumnisurvey@lrwonlinesurvey.com.
  • Links to unsubscribe go to click.skem1.com.
  • The survey itself is at survey.bz.

It all looks pretty fishy/phishy.

Nowhere are there any links to umich.edu.

Also, I happen to know that the University of Michigan tends to use Qualtrics for surveys. Why wouldn’t the university send out a Qualtrics survey from a umich.edu e-mail address with umich.edu unsubscribe links instead of a survey.bz survey from a lrwonlinesurvey.com address with click.skem1.com unsubscribe links?

The survey is likely legitimate. The alumni department probably contracted with a research firm, that research firm probably uses a third-party survey software, and they probably use a different third-party service to handle mailing lists.

But I will not be filling out such a survey. You shouldn’t either. And, if you’re in the business of creating surveys or hiring companies to create surveys, you should think about these factors. Why create something that looks this suspicious?

I’ve always said that survey results automatically exclude those who don’t have time to waste on surveys (this one suggested it would take 18 minutes to complete!), but now it seems they also exclude anyone with a mind for security and privacy.

iptables and deleting/replacing entries

Whenever I have to reboot my modem [sic] at home, I typically get a new IP address from my ISP.

When that happens, I need to update iptables to allow my new address to connect to the SSH port (port 22) of my jump box (which, fortunately, I have access to from another IP address):

iptables -A INPUT -p tcp -m state --state NEW -s [new IP address] --dport 22 -j ACCEPT

But I don’t want to leave the old entry. How to get rid of it?

The delete (-D) and replace (-R) options require a line number from the chain (e.g. the INPUT chain). To find the line numbers:

iptables -L INPUT --line-numbers

To delete the existing rule and add the new rule:

iptables -D INPUT [line number]
iptables -A INPUT -p tcp -m state --state NEW --dport 22 -s [new IP address] -j ACCEPT

To replace the existing entry:

iptables -R INPUT [line number] -p tcp -m state --state NEW --dport 22 -s [new IP address] -j ACCEPT

Save the updates so they are persistent:

iptables-save > /etc/iptables/rules.v4

(That’s the location for Debian and Ubuntu. This may be different for your distribution.)

UPDATE rows with values from a table JOIN in Oracle

Example use case: I have a database that contains a table of contacts (contact) and table of e-mail addresses (email), joined on contact.id = email.contact_id. I just found out that Example Conglomerate acquired Osric Publishing’s Oracle consulting business, and so I need to update my contacts database so that all of the Oracle consultants who had @osric.com e-mail addresses now have @example.com e-mail addresses.

How can I change just the affected addresses in the contact database, assuming the username portion of their e-mail addresses remains the same?
Continue reading UPDATE rows with values from a table JOIN in Oracle

3 ways to iterate over lines of a file in Linux

Frequently I need to run a process for each item in a list, stored in a text file one item per line: usernames, filenames, e-mail addresses, etc. Obviously there are more than 3 ways to do this, but here are 3 I have found useful:

Bash
sh prog1.sh list.txt

Source: prog1.sh

while read line
do
    echo $line
done < $1

4 lines. Not bad.

Perl
perl prog2.pl list.txt

Source: prog2.pl

while(<>) {
    print `echo $_`;
}

3 lines. Pretty good.

Perl -n
perl -n prog3.pl list.txt

Source: prog3.pl

print `echo $_`;

1 line! The -n switch basically wraps your Perl code in a loop that processes each line of the input file. I just discovered this while flipping through my 17-year-old copy of Programming Perl (link is to a newer edition).

I really like this method because you can write a script that processes a single input that could easily be reused by another script, but can also easily be used to process an entire list by adding just the -n switch. (There’s also a similar -p switch that does the same thing, but additionally prints out each line.)

I should note that in the examples above, I am using echo as a substitute for any command external to the script itself. In the Perl examples, there would be no need to call echo to merely print the contents of the line, but it’s a convenient stand-in for a generic command.

As suggested by a comment on a previous post, I have made these examples available in a git repository: iterate over lines.