About a week ago, my wife and I were visiting friends, and I found myself in a room with 3 neuroscientists. The topic of statistics came up, and I managed to insert into conversation my small triumph in analyzing the click-thru study and determining both a confidence interval and the number of tests that would need to be run in order to have a meaningful confidence interval. “Sure,” one of the scientists says, “but what you should really do instead is a chi-square test for goodness-of-fit.”

I had no idea what that meant, or even how it was spelled (I was thinking Kai at first instead of chi), but I found a description in my statistics textbook.

In the original post, there were 6 different banner advertisements, which varied only slightly. Each one was run for 30,000 impressions, which resulted in 6 click-frequencies:

The chi-square test calls not just for observed values, but also for expected values:

For the expected value, I used the mean of all 6 ads, 93 clicks per 30,000 ad impressions. (I don’t know if this is the best value to pick, but it falls in line with the textbook examples.)

I got a chi-square value of 5.42. I wasn’t positive in this case if there were 5 degrees of freedom, or 4, but in either case, using a lookup table, it suggested that the p-value was greater than 0.1, which further suggests that the differences in clicks may have been due to chance, rather than differences in ad design.

My wife uses several stats software packages, so we entered my data into GraphPad Prism and ran a chi-square test there. It came up with a chi-square value very close to mine (though not identical–I’ll have to check my figures again), and a p-value of 0.37. That value, if I interpret things correctly, means there was a 37% chance that the results were due to random chance.

I realized part-way through this exercise that there is a way to trick people into believing the truth without any analysis at all, though. Instead of focusing on the clicks, focus on the instances where people didn’t click. Sure, if one ad gets 81 clicks and another gets 108 clicks, one seems clearly superior to the other. If you instead compare 29,919 non-clicks to 29,892 non-clicks, the difference seems trivial.

I am more satisfied with my result, though. We should be skeptical when people try to convince us that such small differences are significant over so few tests.

Although we made animated ads, we avoided anything that blinked. There were no monkeys to punch.

Click-Through Rate (or Click-Thru Rate or CTR) was a key measurement of an ad’s success. Although at first we would sometimes see click CTRs between 1-2% (meaning that an ad was clicked between 10 and 20 out of every 1000 views, or impressions), as online advertising proliferated, and as our systems got better at filtering out false impressions and clicks from various robots, crawlers, and spiders, CTRs trended much lower: 0.25% suddenly looked good, and 0.10% was not uncommon in some cases. That’s 1 click for every 1000 impressions.

That’s why we were insanely interested in a blog post we found, now presumably lost to the ages, that ran a set of 6 banner ads, which varied only slightly, and analyzed the results to determine what aspects of the ads could improve CTRs. Did including the phrase “click here” really help? If the words “click here” were in blue and underlined, like a typical web link, would that improve the CTR?

They ran 30,000 impressions of each ad. I don’t recall if they set a frequency cap (wherein you limit the number of impressions a specific viewer sees); I don’t believe they did. I don’t recall exact details of each ad or the results, but it looked something like this:

1. Ad for amazing offer: 0.27% CTR
2. Same ad with “click here” in black on the left side of the ad: 0.28% CTR
3. “Click here” in black on the right side of the ad: 0.30% CTR
4. “Click here” in blue on the right side of the ad: 0.32% CTR
5. “Click here” underlined in blue on the right side of the ad: 0.33% CTR
6. “Click here” in blue, inside a button on the right side of the ad: 0.36% CTR

Naturally, the result of this was that all of our ads soon had a gray button in the lower-left corner with the words “Click Here” in it, underlined and in blue:

Example banner ad

I was a bit skeptical, though. Could we really say from 30,000 impressions that a 0.33% CTR is significantly different from 0.36%? It’s a difference of 9 clicks. Could that be attributed to random chance?

I don’t have a background in statistics, so I asked my father, a scientist. He handed me a 1000-page epidemiology textbook and said–and I love this part–that a banner ad click is a lot like a disease state: an individual either has the disease (a click) or does not have the disease. Needless to say, I didn’t make a lot of headway into the world of epidemiology, but the question still troubled me.

Now I am taking an introductory class on statistical analysis, and although my analysis may oversimplify things greatly, I think it is safe to say that we should not have concluded that every ad needed a button-like box with the words “click here” in blue in the lower-right corner.

If we look at any one of the banners in isolation, the CTR is really just a sample mean proportion [thanks to Patrick for the correction]. We could run millions of impressions of the same banner–would it have the same CTR? What is the standard error? To find the confidence interval for the best performing ad, we can run it through this equation:

p +/- z*sqrt((p(1 - p))/n)
p = 0.0036, z = 1.96 (for 95% confidence), and n = 30,000.

The result? 0.36% +/- 0.07%. We are 95% confident that the true population CTR is somewhere between 0.29% and 0.43%. Well–yikes! I’m 95% confident that our measurement isn’t very precise. When we’re dealing with such low proportions, we could really use more precision. We would need to run a test with more than 30,000 impressions.

What if we wanted to run a test where we were 95% confident that our value was within just one-one hundredth of a percent (0.01%) of the population mean? In other words, CTR% +/- 0.005%? We have an equation for that too:

n = (z^2*p(1-p))/e^2
n = desired number of impressions, z = 1.96 (for 95% confidence), p = 0.0036, e = 0.00005

To be 95% confident that our sample CTR is within 0.01% of the population CTR, we would need to run
5,517,522 impressions.

Although the data presented in that blog post from years ago seemed compelling, I think I was right to be skeptical. As I said, this is based on what I’ve learned from an introductory course on statistical analysis. If you think I’m way off base, feel free to enlighten me in the comments.