Migrating to a new GitHub Enterprise host

I’m moving my VMWare infrastructure from old hardware to new hardware. One of the last guest VMs I’ve waited to move is my GitHub Enterprise (GHE) host. My plan to migrate the system was simple:

  1. Create a new, empty GitHub Enterprise VM on the new VMWare infrastructure
  2. Put the old GHE system in maintenance mode
  3. Take a backup of the old GHE system
  4. Shut down the old GHE system
  5. Start the new GitHub Enterprise VM and select the Migrate option
  6. Restore the backup to the new GHE system using the ghe-restore tool

The installation instructions provided by GitHub are pretty good. To deploy a new GitHub Enterprise image I followed Installing GitHub Enterprise on VMWare.

Of course, no process is perfect. Here are a couple minor points that may save you some time:

Continue reading Migrating to a new GitHub Enterprise host

vCenter 6.x: Unable to deploy template

I had a VM on vSphere 5.5 that I was trying to move to a new ESXi 6.7 host via vCenter.

First I exported the OVF template from vSphere 5.5.

Then in vCenter, on the vSphere 6.7 host, I deployed the OVF template.


The "Deploy OVF template" operation failed for the entity with the following error message.

Unable to deploy template.

The error was vague. Why was it unable to deploy the template? Is there a compatibility issue between 5.5 and 6.7?
Continue reading vCenter 6.x: Unable to deploy template

Roomba 530 doesn’t hold a charge

I bought a refurbished Roomba 530 on Woot in 2009, and it’s been an indispensable part of the household ever since. Sure, the cat hates it, it regularly knocks things over, and it somehow escapes its constraints and sneaks off into other parts of the house, but the point is, it cleans while I do something else.

It’s always a satisfying moment when I start the dishwasher, the washing machine, and Roomba at the same time and shout, “Get to work, robots!” and then leave.

One fateful day in 2018, I spilled some water* and Roomba, hapless as ever, bee-lined straight into it. It stopped. Its light turned red. It sang out some tones in a minor key that fit the rhythm of, “Uh-oh. What have I done?”
Continue reading Roomba 530 doesn’t hold a charge

Postfix: altering the subject line of outgoing messages

Motivation: I’m changing my ticketing system so that messages with a friendlier subject line. Instead of ‘Subject: [SNAFU #1] summary’, I’ll change it to use ‘Subject: [HELPDESK #1] summary’.

Another colleague suggested we use procmail. As unixgeeks.org says:

Is procmail right for me?
Procmail is a serious unix hack. On the other hand, it does a pretty good job.


I think we can do better than a serious Unix hack. I think Postfix can handle this.

How to do it? Continue reading Postfix: altering the subject line of outgoing messages

curl basic auth using base64 encoded credentials

I was trying to access password-protected files via HTTPS using curl. The site required basic auth. For a demo, I created this example:

Username: admin
Password: 123456

It’s trivial to access this interactively via curl:

$ curl -u admin https://osric.com/chris/demo/admin/
Enter host password for user 'admin':

Or programmatically by providing the credentials in the URL:

$ curl https://admin:123456@osric.com/chris/demo/admin/

Or by providing a base64-encoded username:password pair in an Authorization header:

$ curl -H "Authorization: Basic $(echo -n admin:123456 | base64)" https://osric.com/chris/demo/admin/

(Note that echo includes a trailing newline character by default, which we do not want to include in the base64-encoded value. Specify the -n flag to echo to eliminate the trailing newline.)

But I was manipulating files with a Bash script that was being stored in a Git repository, and I didn’t want to store the credentials in the repository. So I stored the credentials in a separate file:

$ echo -n 'admin:123456' > ~/admin-credentials
$ chmod 0600 ~/admin-credentials

Now I can read the credentials from the file:

$ curl -H "Authorization: Basic $(cat admin-credentials | base64)" https://osric.com/chris/demo/admin/

I ran into a problem when I tried to update the credentials file with vi (or vim). Vi automatically inserts an end-of-line (EOL) character, which is not apparent to the user. The base64-encoded value includes the EOL character, and therefore the above command would supply invalid credentials.

To eliminate this in vi, use the following vi commands:

:set binary
:set noeol

Alternately, just overwrite the file with the updated credentials:

$ echo -n 'admin:123456' > ~/admin-credentials

AttributeError: module ‘paramiko’ has no attribute ‘SSHClient’

I have a simple Python 3 script (I’m running Python 3.6.1, compiled from source) that does the following 3 things:

  1. Connects to remote server(s) and using scp to get files
  2. Processes the files.
  3. Connects to another remote server and uploads the processed files.

I moved this script from a soon-to-be-retired CentOS 6 host to a CentOS 7 host, and when I ran it there I received the following error message:

AttributeError: module 'paramiko' has no attribute 'SSHClient'

The line number specified in the stack trace led me to:

ssh = paramiko.SSHClient()

First things first: Google the error. Someone else has seen this before. Sure enough, on StackOverflow I found Paramiko: Module object has no attribute error ‘SSHClient’

But no, that’s not the problem I’m having.

I tried to create the the simplest possible script that would reproduce the error:


import paramiko

def main():
        ssh = paramiko.SSHClient()

if __name__ == "__main__":

I ran it as a super-user and received no errors:

$ sudo /usr/bin/python3 test.py

I ran it as myself, though, and it reproduced the error message. Maybe something with the user permissions?

$ ls -l /usr/local/lib/python3.6/site-packages/
total 1168
drwxr-x---.  3 root root   4096 May 29 14:25 paramiko

Oh! From that you can see that unless you are the root user, or a member of the root group, there’s no way you can even see the files within the paramiko directory.

What’s the default umask on the system?

$ umask

That explains it. Now, to fix it. I could probably just run:

$ sudo chmod -R 0755 /usr/local/lib/python3.6/site-packages/*

That should have been the end of that, problem solved! But in my case, installation of the pip modules had been handled by Ansible. I needed to fix the Ansible tasks to account for restrictive umask settings on future deployments. See the umask parameter in the documentation for the Ansible pip module. I updated the task:

- name: Install specified python requirements
    executable: /usr/local/bin/pip3
    requirements: /tmp/pip3-packages
    umask: 0022

Running the playbook with that task, I received an error:

fatal: [trinculo.osric.net]: FAILED! => {"changed": false, "details": "invalid literal for int() with base 8: '18'", "msg": "umask must be an octal integer"}

Another helpful StackOverflow post suggested the value needed to be in quotes:

- name: Install specified python requirements
    executable: /usr/local/bin/pip3
    requirements: /tmp/pip3-packages
    umask: "0022"

Now the playbook runs without error, but it doesn’t change the existing permissions. The task does nothing, since the Python pip modules are already installed. To really test the playbook, I need to clear out the existing modules first.

Warning: this breaks things!:

$ sudo rm -rf /usr/local/lib/python3.6/site-packages/*

I tried running the playbook again and received this error message:

stderr: Traceback (most recent call last):\n  File "/usr/local/bin/pip3", line 7, in <module>\n    from pip import main\nModuleNotFoundError: No module named 'pip'\n

I wasn’t able to run pip at all:

$ pip3
Traceback (most recent call last):
  File "/usr/local/bin/pip3", line 7, in <module>
    from pip import main
ModuleNotFoundError: No module named 'pip'

Clearly, I had deleted something important! I reinstalled Python from the gzipped source tarball for Python 3.6.1 (newer versions available from Python source releases) and then everything worked as expected.

NTP checks with icinga2

On my new Icinga2 monitoring host, I am slowly adding additional service checks to achieve parity with my existing Nagios monitoring. Next on my list, implementing NTP checks. The first step was to add a new service check to the Icinga2 configuration:


apply Service "ntp_time" {
  import "generic-service"
  check_command = "ntp_time"
  assign where host.vars.os == "Linux"

The service check produced an error, as seen in the icingaweb2 interface:

execvpe(/usr/lib64/nagios/plugins/check_ntp_time) failed: No such file or directory

Oh! I don’t have the appropriate Nagios plugin installed on the Icinga2 host.

sudo yum install nagios-plugins-ntp

The NTP service check now reports OK on some hosts, but on other hosts I get a different error:

CRITICAL: No response from NTP server

The hosts that did not receive a response are all using chronyd. I edited /etc/chrony.conf and added:


And restarted chronyd:

systemctl restart chronyd

Now all but one host reports OK. The last remaining host to show an error? The Icinga2 host itself!


Another chronyd restart, and the NTP service on all hosts reports OK.

NRPE: Unable to read output

This one was a real facepalm moment, but I thought I’d share in case anyone else runs into the same thing.

I’ve been working on migrating from Nagios to Icinga2. One of the services I monitor is whether or not a given host has any available yum updates. This service, which I label check_yum, worked on all my hosts except for the Icinga2 host. All the other services monitored on that host were working, but check_yum returned an error:

NRPE: Unable to read output

I tried running the test manually on the Icinga2 host:

/usr/lib64/nagios/plugins/check_nrpe -H localhost -c check_yum
NRPE: Unable to read output

I checked to make sure NRPE was listening, in this case via xinetd:

lsof -i

I checked the service definition to see what script/plugin NRPE runs:

cat /etc/nrpe.d/check_yum.cfg
command[check_yum]=/usr/lib64/nagios/plugins/check_updates -w 0 -c 10 -t 60

I tried to run that manually and…the file /usr/lib64/nagios/plugins/check_updates did not exist.

I installed the corresponding yum package:

sudo yum install nagios-plugins-check-updates

Now it works! It was a reminder to myself to check the basics before trying to troubleshoot network issues.

Re-bind host to FreeIPA

The sudo command on one particular FreeIPA-bound host was taking an exceedingly long time to run. And when it finally ran, it would not accept my current password, but rather my previous password — somehow still cached on the system. It was a strange problem.

Instead of trying to figure out exactly why it was happening, I decided to remove & re-bind the host to my FreeIPA domain.
Continue reading Re-bind host to FreeIPA

Javascript Unit Tests with TravisCI and QUnit

I wanted to make an update to my Bingo Card Generator. Previously I’d added automated tests for JSHint via TravisCI to test for code formatting, but now I wanted to actually test code functionality. I decided to add a simple unit test using QUnit.

I made a lot of mistakes trying to get this to work (probably because I didn’t read the documentation!), so I’ve included all my steps and missteps below in case they are useful to someone else:
Continue reading Javascript Unit Tests with TravisCI and QUnit