BMA Colorado Newsletter

BMA Marketing Mirror Banner

Click here to subscribe to the BMA Marketing Mirror RSS Feed

8/4/2009 - Let The Results Speak For Themselves...Even If You Don't Like What They're Saying

By Tom Pearson, Senior Marketing Communications Specialist, Bulldog Solutions

Message testing is essential to wringing the most value from your e-mail marketing campaigns. Although the experts often advise multivariate testing, the creative team here at Bulldog has found even A/B testing can deliver valuable insight. The world is a pretty noisy place right now, and anything you can do to get a leg up on understanding what's resonating with your prospects is worth pursuing.

Among the tricks I've learned about effective message testing is to let the results speak for themselves, even if you don't like what they’re saying. It's easy to let assumptions sneak into your "objective" testing. Yes, you generally need to work from a hypothesis. But it's important not to let your preconceived notions become the "right answer" by default. Keep an open mind and you'll get more valid insights—or, at least, be able to steer away from obviously invalid ones.

"In the beginner's mind there are many possibilities, but in the expert's there are few." I doubt Zen master Shunryo Suzuki had e-mail marketers in mind when he coined this phrase, but it's spot on, nonetheless. Here are some examples of message testing we've done here at Bulldog that produced results we didn't expect.

Example 1: The Detailed Subject Line

We recently had the opportunity to do an A/B test for one of our clients, and I suggested that we test two different subject lines and their effect on conversion to registration (measured as the number of people who actually registered for a Webinar divided by the number who received the invitation).

At the time, I was enamored of the idea that a longer, more detailed subject line would achieve a higher conversion rate to registration. I theorized that the details might turn off people who were not interested in the topic, thus affecting open rates, but would appeal to those more likely to register. By contrast, a less detailed subject line might see a respectable open rate but fewer registrants.

I'll cop to being pretty confident as we waited for the results to trickle in. As you can guess, the initial numbers showed the issue wasn't cut and dried.

Subject line A: 90 characters, more specific.
Open Rate: 41.43%
CTR: 1.79%
Conversions: .38%

Subject line B, 60 characters, less specific.
Open Rate: 49.30%
CTR: 6.11%
Conversions: 2.42%

Certainly, factors such as deliverability played into these numbers. Even in the best case, we would have done additional testing before striking a conclusion. What was interesting to me about the test is that sensation of wanting to defend my theory to the client even after the data showed I might have been wrong. After spending so much time and effort setting up a test, it's easy to develop that "expert mind," the kind that might even interpret the results to fit the hypothesis.

Example 2: Incentives

We're frequently asked by clients about the efficacy of incentives, from material giveaways like a white paper or a book to drawings for iPods, trips and other items that aren't related to the content. Clients want to know, does including an incentive really impact registration?

In preparation for a Webinar on a marketing thought leadership topic, we did an A/B test offering to enter registrants in an iPod drawing. To our surprise, the iPod test group didn't register at a significantly higher rate than the non-iPod test group. After another group of tests, we concluded that a white paper or briefing document is a better draw for people who are interested in a specific topic.

Here's where it gets interesting, though. Months later, we conducted some testing in which we added an iPod nano incentive to our Marketing Watchdog Journal, offering to enter subscribers into a drawing for referring colleagues to the newsletter. There was some hesitance about using an iPod as an incentive due to the previous findings. But we kept open minds and implemented the test.

And in that case, the iPod worked.

Refer a Colleague iPod Promotion Results:
Month 1: 1,280% increase in referrals
Month 2: 20% increase in referrals
Month 3: 125% increase in referrals

We concluded that the success of the incentive is related in part to the tone of the call to action. A non-material incentive was more successful with a low-value call to action (the newsletter). For a higher value call to action (the Webinar), a material incentive related to the topic was a better draw.

Lessons Learned

The essential mindset for testing boils down to this: Pour your heart and soul into the planning, and then be prepared to let go. Be smart about formulating your hypothesis, plan carefully, execute flawlessly and consistently shoot down the expectations that crop up while you're parsing the incoming data.