Direct mail ROI measured with your own holdout test

Direct mail ROI compares the profit a mailing adds with the cost of running it. To estimate that added profit, compare customers you randomly selected to receive mail with a similar group you held out.

Set up the comparison before you send. Keep the audience, observation period and calculation in your campaign record so you can explain how you reached the result.

Direct Mailer's direct mail analytics gives you mailing activity, spend and returns. You combine that information with sales from your own store or CRM. Direct Mailer doesn't match orders or create holdout tests.

Calculate direct mail ROI from incremental profit#

Use the same currency throughout your calculation. For a Canadian campaign, that may mean converting all campaign expenses and sales to CAD before comparing them.

The formula is:

ROI = (incremental contribution before campaign cost − campaign cost) ÷ campaign cost

Format the result as a percentage if that's how your business reports ROI.

Incremental contribution is the extra revenue associated with the mailing after allowing for the costs of fulfilling those sales. Campaign cost includes the cost of preparing and sending the mailing.

Keep those categories separate. If you subtract printing and postage in campaign cost, don't also subtract them when calculating the contribution from sales.

Use the direct mail ROI calculator to explore an estimate before sending. Enter your own assumptions about response, conversion, order value, margin and mailing cost. The result helps you plan; it doesn't establish what a campaign caused.

Keep all campaign costs in the calculation#

Postage is part of the campaign budget. Your artwork, list preparation, platform fees and any work purchased specifically for the campaign may also belong in it.

Use actual charges where available and document how you allocate shared expenses. If an existing design is reused across mailings, choose a consistent way to account for its cost.

Use revenue after discounts and refunds consistently. If your margin figure already accounts for a cost, don't deduct that cost again.

Understand what matchback tells you#

Matchback compares the people on your mailing list with your customer or order records. It can show which recipients purchased during your observation period, even when they didn't mention the postcard.

Those purchases are useful to know about. They don't show how many would have happened without the mailing.

A previous customer may already have planned to reorder. An email or another offer may also have influenced the sale. Crediting every matched order to the postcard can overstate the mailing's contribution.

A randomized holdout gives you a comparison. Customers in both groups can continue receiving your usual marketing, while the test group receives the additional mailing. Keep that difference clear throughout the test.

Prepare a direct mail holdout test#

Start with the audience you genuinely intend to mail. Remove failed addresses and do-not-mail contacts before splitting the list.

Then randomly assign eligible customers to mailed and holdout groups. Save the assignment in your own records and send only to the mailed group.

Choose the assignment unit carefully. If several contacts share a household, assigning them together can help avoid sending a postcard to someone living with a person in the holdout.

Write down the offer, the sales you will count and the end of the observation period before launch. Keep other marketing treatment consistent across the groups as far as you can. Record any differences you couldn't control.

An offer that expires quickly and a service with a long buying process may need different observation periods. Use your buying cycle to choose one, then apply it equally to both groups.

Our Shopify direct mail guide applies this approach to a customer win-back campaign. The audience split and order comparison happen in your own data.

Choose a sample size before committing the mailing#

A fixed holdout percentage doesn't tell you whether a test can detect a useful difference. You need a baseline purchase rate, the smallest improvement worth detecting and decisions about statistical confidence and power.

Confidence describes how you handle uncertainty in the estimate. Power describes the test's ability to detect an effect of the size you planned for. Smaller effects and less predictable outcomes generally need more observations. NIST's sample-size guidance explains why the assumptions must come first.

Ask an analyst to calculate the required mailed and holdout group sizes for your chosen outcome. Purchase rate and contribution per customer are different outcomes and may require different calculations.

If your eligible audience is too small, the mailing can still help you learn about preparation, address quality and customer response. Keep that learning separate from a confident claim that the campaign increased profit.

Compare the groups fairly#

Groups don't have to contain the same number of people. Compare contribution per assigned customer rather than comparing raw sales totals.

Calculate:

Contribution per customer = total contribution in the group ÷ customers assigned to the group

Then:

Estimated incremental contribution = (mailed contribution per customer − holdout contribution per customer) × mailed group size

Subtract campaign cost and divide by that cost to estimate ROI. This scales the holdout comparison to the audience you mailed.

Use the saved assignment as your starting point. Don't remove people from a group after seeing whether they bought, and don't count only recipients who responded. Record returned and failed mail so you can investigate the mailing without quietly changing the original comparison.

Ask your analyst for an uncertainty interval around the result. A positive estimate with wide uncertainty needs a different decision from a result that consistently clears your cost threshold.

Treat published benchmarks as context#

A “good ROI for direct mail” depends on what the campaign needs to achieve after its costs. Reaching past customers and introducing a business to prospects are different jobs.

Before borrowing a direct mail response rate, check what the source counted as a response, who received the mail and when the research was conducted. An enquiry rate cannot stand in for a purchase rate.

Also check whether a published return uses revenue or profit, and whether it accounts for purchases that would have happened anyway. Those definitions can change the meaning of a headline result.

Keep a record you can use next time#

Save the final audience assignment, artwork, offer, campaign expenses, mailing dates, observation period and sales calculation together.

Check mailing progress before interpreting an early sales result. Direct Mailer shows Mailed when the item reaches Canada Post, with an estimated arrival. It doesn't confirm when an ordinary postcard or letter arrives.

At the end of your chosen period, record the estimate, its uncertainty and what you would change in the next mailing. Keep the same definitions when comparing campaigns so a change in the calculation doesn't look like a change in performance.

Questions about direct mail ROI#

Is revenue divided by mailing spend the same as ROI?#

That calculation is return on ad spend when it uses attributable revenue and advertising spend. A profit-based ROI calculation accounts for the cost of making the sales as well as the campaign cost. Label the measure you report.

Can I measure a campaign without a holdout?#

You can track responses, match customers to purchases and compare spending with sales. Describe the result as an attribution estimate and state its limits. Without a suitable comparison, you can't assume every matched sale was caused by the mailing.

Should I stop measuring once the campaign breaks even?#

Use the observation period you chose before launch. Stopping when a result looks favourable changes the test. Complete the planned analysis and account for refunds or other adjustments using the same rules for both groups.