Migrating to a new GitHub Enterprise host

I’m moving my VMWare infrastructure from old hardware to new hardware. One of the last guest VMs I’ve waited to move is my GitHub Enterprise (GHE) host. My plan to migrate the system was simple:

1. Create a new, empty GitHub Enterprise VM on the new VMWare infrastructure
2. Put the old GHE system in maintenance mode
3. Take a backup of the old GHE system
4. Shut down the old GHE system
5. Start the new GitHub Enterprise VM and select the Migrate option
6. Restore the backup to the new GHE system using the ghe-restore tool

The installation instructions provided by GitHub are pretty good. To deploy a new GitHub Enterprise image I followed Installing GitHub Enterprise on VMWare.

Of course, no process is perfect. Here are a couple minor points that may save you some time:

When I started the new GitHub Enterprise VM and connected to the console. Once I configured networking, the console provided the next step:

Visit http://192.168.0.123/setup to configure GitHub Enterprise.

However, that didn’t work. It directed me to a blank page at http://192.168.0.123/ssl_warning.html.

The answer? Specify https instead, e.g.:

Visit https://192.168.0.123/setup to configure GitHub Enterprise.

I added the appropriate SSH key from a system where I had the GHE backup and restore tools installed, and then selected the Migration option. It prompted me to run ghe-restore:

$ ./ghe-restore 192.168.0.123
Checking for leaked keys in the backup snapshot that is being restored ...
* No leaked keys found
No ECDSA host key is known for [192.168.0.123]:122 and you have requested strict checking.
Host key verification failed.
Error: ssh connection with '192.168.0.123:122' failed
Note that your SSH key needs to be setup on 192.168.0.123:122 as described in:
* https://enterprise.github.com/help/articles/adding-an-ssh-key-for-shell-access

At first I thought I could just add the key to my known_hosts file:

$ ssh-keyscan -t ecdsa 192.168.0.123

But ssh-keyscan exited with an error code and returned no output.

Eventually I decided to disable StrictHostKeyChecking in my .ssh/config file for the target IP address:

Host 192.168.0.123
	StrictHostKeyChecking no

(Just remember to remove that entry when done! StrictHostKeyChecking is an important security control.)

I should also note that the Running migrations step took forever. I often wondered if it was still working. The first three steps were almost instant, but Running migrations goes on for hours:

1. Preparing storage device ✔
2. Updating configuration ✔
3. Reloading system services ✔
4. Running migrations
5. Reloading application services

I tried visiting the regular (i.e. non-admin interface) and it gave me the following message:

Migrating...
Please be patient, this may take a long time. Please do not reboot or shutdown the appliance until this has completed.  
Performing hookshot database migrations.

How long is a long time? What is a hookshot database migration? Who knows, be patient.

The console shows these messages over and over and over:

19:59:04 elasticsearch: Starting Elasticsearch...
19:59:04 elasticsearch: Started Elasticsearch.
19:59:12 elasticsearch: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
19:59:12 elasticsearch: Unit elasticsearch.service entered failed state
19:59:12 elasticsearch.service holdoff time over4, scheduling restart.
19:59:12 cloud-config: Cannot add dependency job for unit could-config.service, ignoring: Unit cloud
19:59:12 elasticsearch: Stopping Elasticsearch...

The reason was that I missed an important step. With the upgrade from 2.13 to 2.14, you need to upgrade the Elasticsearch indices (see Migrating Elasticsearch indices to GitHub Enterprise 2.14 or later).

I started from the beginning again, adding the above Elasticsearch migration step just after putting GHE into maintenance mode. Everything else proceeded smoothly, and in a much more reasonable amount of time.

It would have been nice if the GitHub Enterprise migration software had been able to identify and notify me of this problem instead of going into an infinite loop. An infinite loop that occurs simultaneously with a Please be patient, this may take a long time message. I hope that by including my misstep here, I might save someone else a few minutes — or hours — of frustration.