Why Holdout Testing is the Key to Measuring the Effectiveness of Your Direct Mail

Even if you know your response rate, do you know the incremental lift of your direct mail over your other marketing efforts? While the world of data science and new technology constantly evolves, one foundational element of best-practice direct marketing remains solidly in place: holdout testing, which allows savvy marketers to measure the incremental lift and ROI of their marketing efforts—and to help generate valuable insights for ongoing optimization.

A Simple Proposition

The basic concept is simple. There is a population of customers and/or prospects that you can market to via direct channels, whether by direct mail, telemarketing, email, or even programmatic advertising. You randomly divide that population into two test groups: One that receives the mailing (General Mail File) and one that does not receive the mailing (Creative Holdout). After the campaign has been completed, you measure the response and sales of both test groups.

While the Creative Holdout group did not receive the mail offer, you will most likely observe sales activity driven by other media channels, such as TV, radio, email, display advertising, social, paid search, or even organic search. This holdout group constitutes a baseline against which you can measure the incremental lift of your direct mail marketing effort.

In our example below, we are selling a product for $500. The Creative Holdout group generates a baseline 0.30% response rate, yielding $1.50 in sales per person. The General Mail File group response rate is 0.80%, generating $4 in sales per person. The higher response rate translates to an incremental $2.50 lift in sales per person.

Consider Two Holdout Cells in Your Test Design

In addition to the Creative Holdout, best practice is to also have a separate Data Holdout File (test cell 2 below). This cell receives mail and is the same size as the Creative Holdout cell. The Data Holdout File allows for estimating the often-observed drop in response rate from test cell to rollout. It is important that the data in each cell have the same characteristics.

When marketing expenses are factored in, you can then calculate the incremental ROI of the mailing. It is important to use the rollout cost per household name in evaluating the results. For those less familiar with direct mail, sometimes a smaller test is more expensive to produce. Efficiencies are gained as mail quantities increase.

In the example below, $0.60 is used as a rollout cost to mail per household for illustrative purposes.

Our conclusion from this example would be that mailing produced 2.27 times more net revenue per household than not mailing. Also, the responders to direct mail campaigns can often be more valuable over time. As your direct mail campaign experience accumulates, a lifetime value analysis of direct mail responders can also help validate this direct marketing channel.

How Big Should My Holdout Testing Group Be?

While there is no one correct answer, one fundamental statistical truth prevails: as the volume of data increases, so too does your ability to generate statistically significant insights that will allow you to further optimize your marketing program.

If your only goal is to measure the overall incremental lift of the program, then the holdout group must have the minimum quantity to achieve statistical significance, and that quantity will be determined by the characteristics of the specific metric you wish to measure.

However, you can go much deeper with more data. Differences in No Mail/Mail results by dimensions, such as region or product, could generate valuable insights for optimizing future mailings. For example, a catalog company might discover that the mailings do not provide any lift for a best-selling product and decide to swap in another one that needs support. Or perhaps there is a very strong lift in a certain region, an insight that could provide the impetus for increased investment in future campaigns.

Finding the Right Balance

In the real world, there is always pressure to maximize sales, and withholding “good names” from a campaign is counter to maximizing short-term financial results. Creative and list tests also have a cost, as most fail to beat the control.

But to be successful over the long term, direct marketing programs must have a strategic testing plan in place that is supported by rigorous analytics. If the tests are properly planned, executed, and analyzed, the long-term value of the testing insights will far exceed the short-term cost.

So, if you’re not doing holdout testing today in your current campaigns, you should give it a try—who you don’t mail to could teach you a lot.

If you’re interested in trying out direct mail and adding it to your marketing mix, then drop us a line. We’re standing by to answer any questions you might have, and most importantly, to help you get your mail opened.