Let’s pretend you and I are in a bake-off to see who makes the best brownies. I take my brownies to 10 random people at work and 6 of them loved it while the other 4 did not. The same day, you knocked on every door in a 3-mile radius and recruited 100 neighbors (both friends and strangers) to try your brownies. Half of them liked it and the other half did not. Hakim, my little brother who has been dealing selling brownies for the past month, also decided to join our competition and recruited 100 strangers to try his brownies, winning over 70% of those people with his secret recipe (psst… the secret is weed).

This is the setup Bad Brain refers to as “The Brownie Problem” with our clients but in statistics, it is the Law of Large Numbers (LLN) and regression to the mean at-play (along with other factors like “first-to-market”, selection bias, technological determinants, and threat of new entrants — all subjects we have experiential knowledge in navigating and will cover in later posts. I’ll try and remember to link them here).

The contest was really between you and me, so we will focus on those numbers first. If we look at the conversion rate, I won — I have a 60% yum rate and you only have 50%. That does not seem fair, does it? I only had 10 testers and you had 100 — you had to try and please a lot more people than I did. Somewhere along the internet, you learned that confidence intervals are ripe for ranking formulas and help mitigate risks associated with ranking rates with a large variance in the sample sizes. Rather than asking me to go and find another 90 people, which would take forever because I’m an introvert, you offer to input our numbers into a confidence interval formula that would essentially put us on equal footing. Just to prove that your formula is fair, you show me that Hakim would have still been the clear winner. Fine, maybe you do have the more likable brownie.

The Brownie Problem consistently occurs in your advertising reports. Here is a real-life example (we have omitted our client’s name and other identifiers):

The client above is a premium-priced product in the health and wellness space developing a category with just one other competitor. The language around both the product and the category require exhaustive testing, as market development must precede product marketing when selling innovation. The client, typically inclined to look at straight conversion rates or Cost Per Order (CPO) when allocating their budget, trusted the platform to serve the “best performing creative”, as the platform promises. 

The platform was not serving the best performing creative and the impression count favored a poorer performing ad. We needed to force the platform to push the best messaging, but with a large variance across the impression counts, we found ourselves in the thick of The Brownie Problem. Just as I demonstrated above in The Brownie Problem, we placed the reports on a confidence interval to even-out the playing field for the creative. We found the platform was helpful with revenue generation but needed spending boundaries to function optimally where creative was concerned. If you scan the copy, you will find the obvious: with a prestige item, lowered pricing will always drive sales, closely following the measurement of price elasticity of demand. That’s fine but we needed something more substantive to understand what effectively promoted the product to retention and acquisitional audiences.

We found that harping on the 10% loyalty discount for retention audiences, though the platform served that copy the most, performed worse than copy that reminded the target of the product’s most revolutionary quality. With acquisitional audiences, it was the measurable and pragmatic idea behind the product, the simple promise of a “5-day reset”. If the client had continued looking solely at conversion rates and CPOs, completely ignoring LLN’s and regression to the mean’s impact on reports, they would have continued pushing always-on discounts to retention audiences and the name of the science to acquisitional audiences (the name is a mouthful though plain English). The findings allowed us to optimize a creative refresh that improved ROAS by 35% on a $2.5MM budget and increased the sales velocity with preened, focused ads fit for the platform, audience, and brand.

Here is another client example that demonstrates how confidence intervals provide a clearer picture. 

For this client, we are currently testing landing for sales conversions better when used as an ad landing page. If we look at the conversion rates, /science appears the better option, but when we introduce a confidence interval into the formula, we see that the two pages are rather equal in terms of converting power. 

We encourage you to check all your rates on a confidence interval. If you don’t know which formula is right for you, hit us up. □