About a week ago, my wife and I were visiting friends, and I found myself in a room with 3 neuroscientists. The topic of statistics came up, and I managed to insert into conversation my small triumph in analyzing the click-thru study and determining both a confidence interval and the number of tests that would need to be run in order to have a meaningful confidence interval. “Sure,” one of the scientists says, “but what you should really do instead is a chi-square test for goodness-of-fit.”

I had no idea what that meant, or even how it was spelled (I was thinking Kai at first instead of chi), but I found a description in my statistics textbook.

In the original post, there were 6 different banner advertisements, which varied only slightly. Each one was run for 30,000 impressions, which resulted in 6 click-frequencies:

The chi-square test calls not just for observed values, but also for expected values:

For the expected value, I used the mean of all 6 ads, 93 clicks per 30,000 ad impressions. (I don’t know if this is the best value to pick, but it falls in line with the textbook examples.)

I got a chi-square value of 5.42. I wasn’t positive in this case if there were 5 degrees of freedom, or 4, but in either case, using a lookup table, it suggested that the p-value was greater than 0.1, which further suggests that the differences in clicks may have been due to chance, rather than differences in ad design.

My wife uses several stats software packages, so we entered my data into GraphPad Prism and ran a chi-square test there. It came up with a chi-square value very close to mine (though not identical–I’ll have to check my figures again), and a p-value of 0.37. That value, if I interpret things correctly, means there was a 37% chance that the results were due to random chance.

I realized part-way through this exercise that there is a way to trick people into believing the truth without any analysis at all, though. Instead of focusing on the clicks, focus on the instances where people didn’t click. Sure, if one ad gets 81 clicks and another gets 108 clicks, one seems clearly superior to the other. If you instead compare 29,919 non-clicks to 29,892 non-clicks, the difference seems trivial.

I am more satisfied with my result, though. We should be skeptical when people try to convince us that such small differences are significant over so few tests.

Although we made animated ads, we avoided anything that blinked. There were no monkeys to punch.

Click-Through Rate (or Click-Thru Rate or CTR) was a key measurement of an ad’s success. Although at first we would sometimes see click CTRs between 1-2% (meaning that an ad was clicked between 10 and 20 out of every 1000 views, or impressions), as online advertising proliferated, and as our systems got better at filtering out false impressions and clicks from various robots, crawlers, and spiders, CTRs trended much lower: 0.25% suddenly looked good, and 0.10% was not uncommon in some cases. That’s 1 click for every 1000 impressions.

That’s why we were insanely interested in a blog post we found, now presumably lost to the ages, that ran a set of 6 banner ads, which varied only slightly, and analyzed the results to determine what aspects of the ads could improve CTRs. Did including the phrase “click here” really help? If the words “click here” were in blue and underlined, like a typical web link, would that improve the CTR?

They ran 30,000 impressions of each ad. I don’t recall if they set a frequency cap (wherein you limit the number of impressions a specific viewer sees); I don’t believe they did. I don’t recall exact details of each ad or the results, but it looked something like this:

1. Ad for amazing offer: 0.27% CTR
2. Same ad with “click here” in black on the left side of the ad: 0.28% CTR
3. “Click here” in black on the right side of the ad: 0.30% CTR
4. “Click here” in blue on the right side of the ad: 0.32% CTR
5. “Click here” underlined in blue on the right side of the ad: 0.33% CTR
6. “Click here” in blue, inside a button on the right side of the ad: 0.36% CTR

Naturally, the result of this was that all of our ads soon had a gray button in the lower-left corner with the words “Click Here” in it, underlined and in blue:

Example banner ad

I was a bit skeptical, though. Could we really say from 30,000 impressions that a 0.33% CTR is significantly different from 0.36%? It’s a difference of 9 clicks. Could that be attributed to random chance?

I don’t have a background in statistics, so I asked my father, a scientist. He handed me a 1000-page epidemiology textbook and said–and I love this part–that a banner ad click is a lot like a disease state: an individual either has the disease (a click) or does not have the disease. Needless to say, I didn’t make a lot of headway into the world of epidemiology, but the question still troubled me.

Now I am taking an introductory class on statistical analysis, and although my analysis may oversimplify things greatly, I think it is safe to say that we should not have concluded that every ad needed a button-like box with the words “click here” in blue in the lower-right corner.

If we look at any one of the banners in isolation, the CTR is really just a sample mean proportion [thanks to Patrick for the correction]. We could run millions of impressions of the same banner–would it have the same CTR? What is the standard error? To find the confidence interval for the best performing ad, we can run it through this equation:

p +/- z*sqrt((p(1 - p))/n)
p = 0.0036, z = 1.96 (for 95% confidence), and n = 30,000.

The result? 0.36% +/- 0.07%. We are 95% confident that the true population CTR is somewhere between 0.29% and 0.43%. Well–yikes! I’m 95% confident that our measurement isn’t very precise. When we’re dealing with such low proportions, we could really use more precision. We would need to run a test with more than 30,000 impressions.

What if we wanted to run a test where we were 95% confident that our value was within just one-one hundredth of a percent (0.01%) of the population mean? In other words, CTR% +/- 0.005%? We have an equation for that too:

n = (z^2*p(1-p))/e^2
n = desired number of impressions, z = 1.96 (for 95% confidence), p = 0.0036, e = 0.00005

To be 95% confident that our sample CTR is within 0.01% of the population CTR, we would need to run
5,517,522 impressions.

Although the data presented in that blog post from years ago seemed compelling, I think I was right to be skeptical. As I said, this is based on what I’ve learned from an introductory course on statistical analysis. If you think I’m way off base, feel free to enlighten me in the comments.

Finally, when I started Apache from the command line I got a more informative error:
(OS 10048) Only one usage of each socket address (protocal/network address/port) is normally permitted. : make_sock: could not bind to address 127.0.0.1:80 no listening sockets available, shutting down

It turns out, I had Skype running, which by default binds to ports 5520, 80, and 443. There are several solutions:

1. Quit Skype and don’t run Skype and Apache at the same time. Not ideal, but it works.
2. Change which port Apache uses. In your Apache2/conf/httpd file, change the line “Listen 80″ to something like “Listen 8888″. Apache should start, but you’ll need to access it via http://localhost:8888 instead of merely http://localhost
3. Change which ports Skype uses for incoming connections. Go to Tools–Options–Advanced–Connection and un-check the box next to “Use port 80 and 443 as alternatives for incoming connections.”

As far as I can tell, the last option has no ill-effect on Skype, although it might if your firewall or router is blocking port 5520.

I ended up installing Apache 2.0 in addition to Apache 2.2, as I was having trouble with mod_perl with the latter. I ran into the same issue there as well, and once I quit Skype I was able to start Apache.

Strangely enough, I now seem to have both Skype and Apache running and both listening on port 80, so perhaps I’ve misdiagnosed the problem completely.

My attempts to install Adobe AIR on the Asus eee 701 (running the default Xandros distro) were foiled several times in spite of following the instructions:

1. Download Adobe AIR
2. Make the AdobeAIRInstaller.bin file executable
3. Run the .bin file as a superuser

I got a nice friendly fail message from the Adobe AIR installer every time.

I found a few relevant forum posts, e.g. Adobe Air Linux won’t install on Eee PC, that suggested memory was an issue. Sure enough, running in Full Desktop Mode with 1440×900 screen resolution (on an external display), I only had about 90MB of 500MB free.

I restarted the eee in Easy Mode and then immediately ran AdobeAirInstaller.bin. Success! (I later found these same instructions on the eee user forums.)

Installing Tweetdeck was trivial at that point: download the .air file, find it in the File Manager, and double-click it. However, when I ran it, it didn’t do anything. At one point I got a message that I was running an unknown desktop, and that Tweetdeck required Gnome or KDE.

I restarted in Full Desktop Mode, and was surprised to find a Tweetdeck icon already on the desktop. I ran it and was prompted to use KWallet, a KDE password manager. I canceled out of that, and found that Tweetdeck opened, but still didn’t do anything.

I tried again, activated the KWallet password manager, and then it worked! Tweetdeck prompted me for my twitter login, I additionally logged in to Facebook, and now I have a mean, lean, social networking machine.

I also feel as if recently my code has taken a big step forward in quality. I’m not sure exactly what did it, but I suspect that it was either trying to write a good written word piece for the first time, or starting to do daily meditation. Either way my code quality has gone up in a visible and measurable way. I feel good about this because I just did another pass through the code base and feel like it’s getting much closer to something that could be licensed with real value.

Installation: I’m hoping to get this as simple as possible. I took the time to write up a custom script “installer_helper.php” that drags the user kicking and screaming through the installation process. Hopefully this is enough, though a few steps still seem to hard. (Rather horrifically, the first one is the worst.)

Publisher: Hopefully this will be so easy that an ape could do it. You can either be logged in as an admin, or visiting the page. If you are logged in and visit a page that doesn’t exsist, you get the option to create the page, by selecting from a list of templates. The new page is automatically added to the master navigation. Each of the templates has areas of text, Images, and other sundry that can be edited by clicking on the little “edit” button next to them. Publishers will never need to touch the web server, or HTML.

Designer: New templates useing the smarty mark-up rules. Their are a few additional calls that can be made to put in editable fields, but for the most part it just straight HTML. The new template is simply deposited in the “view” directory, and then becomes an avalible option for the Publishers. Javascript, Images and CSS can all be put in their respective directories in “webroot”

Developer: If you need to add some actual functionality to the site, Brochure Ape can act as a rough and ready framework as well. Custom PHP code can be written, and put in files in the “controler” directory. At any point if a page is created that matches the name of a controller, it will run both the CMS code, and the controler code. There is also a pretty cool table-object that can be used to deal with specific database tables.

Extender: The ultimate idea is for the core code to be broken out into modules, which have been divided up sanely. Something a little bit more like a micro-kernal approch. At this point it’s just not true, and the extender roll would be a hard and unsupported one. I do however intend to keep on cleaning this up.

This page contains both secure and nonsecure items. Do you want to display the nonsecure items?

I disabled Javascript and the error messages are gone, so I figured the culprit must lie somewhere in the Javascript. One particular file, JavaScriptFlashGateway.js, contained
flashTag += 'codebase="http://download.macromedia.com ... ';

I’m going to see if updating that will address the issue, but I have a feeling there may be other issues. Has anyone else run into this? My Google searches have been coming up empty.

This isn’t really that shocking, since there is a bunch of javascript that gets pushed into the page by debug, what is weird, and a pretty large problem is that once I do this, google maps continues not to load, even after the {debug} tag has been removed. I haven’t really dug into what is going on to carefully yet, but its an interesting issue.

Really lame solution found after the jump

Well, I dug into it, and as it turns out, it was not a problem with the debug statement at all. Rather it was an issue because I had done some line formatting in the javascript that called google maps, and put in some carriage returns in the middle of a string. (that string was producing HTML so I didn’t mind having some whitespace in the output). It turns out that this was what was making the map API puke. Just goes to show… make sure you have isolated the problem correctly!

What I have found is a lot of books that tell me about tools for generating quality work, but ignore the emotional end of the process. One of the few bits of emotional advice that they do turn up repeatedly is that once you let some of a project start to slide into a lower level of quality, all motivation to keep the quality level up drains away in what is typically called “Broken Window Syndrome“. What they don’t talk about is how to recover from it (my own personal holy grail, since I am so often brought in to save messed up projects) or how to feel a sense of quality to start with.

Since I am feeling around in the dark, I thought I would take two blind stabs at raising my quality level. The first idea I am putting into practice is literally sitting down and meditating on quality & craftsmanship before I start to work on a project. This means 15 minutes of cross-legged, closed eye meditation, thinking about the nature of quality, (very much in the style of “Zen and the art of motorcycle maintenance“), and on my own personal visions of a true craftsman at work. This does seem to help.

The second part of this approach that I am taking, is to start a personal project and try to hold it to a very, very high standard of work. The idea behind this is to help develop the habits of quality, and help it become a familiar thing. This will probably be a cakePhp project, which I will most likely write about more here. I’m already looking into getting started with unit testing for it, but more on that later.

If anyone has any tips on emotional self management for programming, or can refer me to a book on it, that would be great as well.