Why Holdout Testing is the Key to Measuring the Effectiveness of Your Direct Mail

09 August 2017 / By Ted Wilkins

Even if you know your response rate, do you know the incremental lift of your direct mail over your other marketing efforts?

Ted Wilkins

While the world of data science and new technology constantly evolves, one foundational element of best-practice direct marketing remains solidly in place: holdout testing, which allows savvy marketers to measure the incremental lift and ROI of their marketing efforts—and to help generate valuable insights for ongoing optimization.


A Simple Proposition

The basic concept is simple. There is a population of customers and/or prospects that you can market to via direct channels, whether by direct mail, telemarketing, email or even programmatic advertising. You randomly divide that population into two groups: One that receives the specific offer (Mail) and one that does not (No Mail). After the campaign has been completed, you measure the response and sales of both groups.

While the holdout No Mail group did not receive the offer, you will most likely observe sales activity driven by other media channels such as TV, radio or web advertising. This holdout group constitutes a baseline against which you can measure the incremental lift of your direct marketing effort.

In our example below, we are selling a product for $500. The holdout No Mail group generates a baseline 0.30% response rate, yielding $1.50 in sales per person. The Mail group response rate is 0.80%, generating $4.00 in sales per person. The higher response rate translates to an incremental $2.50 lift in sales per person. When marketing expenses are factored in, you can then calculate the incremental ROI of the mailing.

How Big Should My Holdout Testing Group Be?

While there is no one correct answer, one fundamental statistical truth prevails: as the volume of data increases, so too does your ability to generate statistically significant insights that will allow you to further optimize your marketing program.

If your only goal is to measure the overall incremental lift of the program, then the holdout group must have a minimum quantity to achieve statistical significance, and that quantity will be determined by the characteristics of the specific metric you wish to measure.

However, you can go much deeper with more data. Differences in No Mail/Mail results by dimensions, such as region or product, could generate valuable insights to optimize future mailings. For example, a catalog company might discover that the mailings do not provide any lift for a best-selling product and decide to swap in another one that needs support. Or perhaps there is a very strong lift in a certain region, which could provide the insight for increased investment in future campaigns.


Finding the Right Balance

In the real world there is always pressure to maximize sales, and withholding “good names” from a campaign is counter to maximizing short-term financial results. Creative and list tests also have a cost, as most fail to beat the control.

But to be successful over the long term, direct marketing programs must have a strategic testing plan in place that is supported by rigorous analytics. If the tests are properly planned, executed and analyzed, the long-term value of the testing insights will far exceed the short-term cost.

So, if you’re not doing holdout testing today in your current campaigns, you should give it a try—who you don’t mail to could teach you a lot.

Ted Wilkins
About The Author

Ted Wilkins

Ted has over 20 years of experience developing and managing marketing programs for top brands across the education, financial services and travel industries.