Recently, I was looking at some of the email metrics for a campaign we ran where I currently work. Since we ran a multivariate test, identifying each success metric was key. The first was easy enough – subject line. We could identify this just by the percentage of opens.
The next thing we were testing was the creative.
At this point, we could have looked at the Click Thru Rate given to us by our email service provider. But this did not sit right with me. Their formula was the total number of clicks divided by the number of sends. But that seems unfair! Why should you compare clicks based on total number of sends? It is not the clicks job to open the email. It is his/her job to get you to click once you have opened it!
I proposed using a new metric:
Clicks / Opens
This seems like it is a much better indicator of how effective your creative really is. And since the number is likely to be much larger, you will really be able to see a trend. For example, rather than a CTR of 1.3 compared to 1.4, which is such a minimal difference you might be inclined to assign this difference to chance, you might see 17% compared to 30%. In that case, it is pretty easy to see which creative won.
The only caveat I can think of right now is those people who use Outlook and have a preview email set. For some annoying reason, ESPs don’t count the preview view as an open, even though it should. Especially when the ESP will count the click from the preview view. But, just because one formula is flawed, does that mean we should continue to skew data to minimize the impact of successful or unsuccessful creative? Since we are trying to understand trends, I do not think so.
As an old boss said to me: “We are about identifying trends. We’re not accounting.” I think when it comes to Open Rates and Click Thru Rates, this is certainly true.
What do you think? How should we evaluate CTRs? Also, do you remember Revenge of the Nerds?
whybegee | Yoav B. Guttman 