Kitchen CI – using the Vagrant driver

I’d previously been using the Docker driver with Kitchen CI and kitchen-ansible to test my Ansible playbooks. I really like using Kitchen CI. Test-driven infrastructure development! Regression testing! It’s great.

There were several reasons I decided to switch from the Docker driver to Vagrant. My target hosts are all either VMs or bare metal servers, so Vagrant VMs more closely resemble that environment. In particular, there are a couple areas where Docker containers don’t perform well for this purpose:

  • Configuring and testing SELinux settings
  • Configuring and testing systemd services

Continue reading Kitchen CI – using the Vagrant driver

Javascript Unit Tests with TravisCI and QUnit

I wanted to make an update to my Bingo Card Generator. Previously I’d added automated tests for JSHint via TravisCI to test for code formatting, but now I wanted to actually test code functionality. I decided to add a simple unit test using QUnit.

I made a lot of mistakes trying to get this to work (probably because I didn’t read the documentation!), so I’ve included all my steps and missteps below in case they are useful to someone else:
Continue reading Javascript Unit Tests with TravisCI and QUnit

Using jshint with Travis CI

I thought I’d give Travis CI a try. It’s a tool that hooks into GitHub and runs automated tests on your code every time you push a commit. I found a straightforward tutorial that basically said I’d need 2 files in addition to my existing code:

  1. .travis.yml
  2. package.json

Simple! I configured Travis CI to run JSHint (a Javascript code linter, similar to JSLint) on the Javascript files in my Simple Steganography project.

I pushed a commit and discovered my files did not pass JSHint. However, I thought they should be. My files were previously configured for JSLint, and JSHint can read JSLint directives. At the top of my Javascript files I had the directives:

bitwise, browser

These indicate that bitwise operators should be allowed, and to assume the code is running in a web browser.

JSHint (via Travis-CI) reported the following errors:

js/decode.js: line 3, col 1, Bad option value.
js/decode.js: line 3, col 1, Bad option value.

The JSHint docs on inline configuration indicate that the formatting should be option: Boolean, so I reformatted the configuration directives:

bitwise: false,
browser: true

JSHint (via Travis-CI) reported several errors regarding bitwise operators:

js/encode.js: line 179, col 37, Unexpected use of '&'.
js/encode.js: line 179, col 42, Unexpected use of '^'.
js/encode.js: line 179, col 53, Unexpected use of '&'.
js/encode.js: line 180, col 51, Unexpected use of '|'.
js/encode.js: line 182, col 51, Unexpected use of '&'.

I changed the JSHint configuration directives to:

bitwise: true,
browser: true

This worked, and my JSHint tests passed! I added a build status image to my GitHub repo:
GitHub build status for Simple Steganography project

The page describing the JSHint Options definitely leads me to believe that enabling the bitwise option would enforce bitwise errors. This is the opposite of the behavior I’m seeing. The problem is either with the documentation, the behavior, or my reading comprehension! I opened a GitHub issue on the JSHint project describing what I experienced.

Test-Driven Programming Assignments

I am enrolled in another graduate course this semester, Theory of Computation. Like last semester, the programming assignments include unit tests (JUnit tests for the Java assignments). One can be very confident prior to submitting an assignment that it is done properly if it passes all the unit tests!

I’ve been interested in unit testing and test-driven development for several years now, but have never put it into practice. It seemed like a good idea in theory, but it required picking a testing framework, installing it, configuring it, and, of course, writing good tests. Without any hands-on experience, it’s a difficult practice to adopt. Getting introduced to unit tests this way lifts nearly all those burdens. On top of that, the advantages are clear: the tests will tell you when you have it right.

I was a little surprised to run into unit testing via coursework, because I’d been under the impression that computer science education focuses a lot on theory, and skips over a lot of the practice. I’ve met a lot of people coming out of computer science programs who, while probably excellent theorists, don’t know heads from tails when you sit them at a terminal. I was happy to see that my coursework included practical knowledge as well.

All the unit tests are provided by the professor. I assume we may write some tests of our own later on–and certainly nothing is stopping us from writing our own tests now–but I do wonder how many students will be prepared to take that next step. From what I understand, writing good tests is the better part of successful unit testing.

Some things I’ve noticed about the provided unit tests:

  • There are a lot of them. The unit tests are half as long the code that passes them.
  • There are many sample values to test the same function, mostly to test specific edge cases. Failure at a specific edge case helps to identify where the code went wrong.
  • In some cases, the tests are within a loop that generates random test data. Although there’s nothing quite like real human input to break your programs, 1000 random tests might help.

In practice, do programmers tend to write their own unit tests? I imagine it would be ideal if you partnered with someone, and you wrote their unit tests, and they wrote yours. It might be easy to overlook an edge case or dismiss something as impossible if you are too involved with the specifications. At the same time, it is probably difficult to write a unit test without spending some time reading the specifications and understanding exactly what it is supposed to do.

Testing for multiple versions of Internet Explorer

Only one version of Internet Explorer (IE) can exist on a single windows installation by default. I had previously used Multiple IE as a way of testing web pages on older versions of IE. This allows you to have IE3, IE4, IE5, IE5.5, and IE6 installed alongside your existing IE7 or IE8 install. You can even run them concurrently.
Internet Explorer Logo
I don’t test pages on anything earlier than IE6 anymore, but IE6 still accounts for more than 5% of my site traffic. Multiple IE basically helps me test both IE6 and IE8 from the same machine. However, there is still the IE7 gap, plus Multiple IE, which is no longer being updated or maintained, can produce some glitchy behavior.

It turns out, though, Microsoft offers some tools to help test their legacy browsers.
Continue reading Testing for multiple versions of Internet Explorer

Testing Tools

A couple of useful testing tools I thought I’d share:

The latter seems a little buggy, but still easier to use than removing and installing different Flash versions yourself.