Permission-based email marketers have the unique ability to quantifiably determine exactly which messages resonate best with their customers.

Despite this explicit advantage, 20 percent of marketers say that they do not perform any testing on their email program, and 55 percent say they perform only some testing on an ad hoc basis with a limited number of elements.

This shows that many businesses still rely on deductive reasoning or occasional, high-level testing to determine the key aspects of their campaigns. This kind of inattention can lead to missed opportunities, increased attrition and diminishing returns, but can easily be avoided through the use of best practices and statistical analysis.

Especially in tough economic times when budgets are tight and resources are constrained, running well-planned, methodical testing of your email campaigns allows you to gain a more detailed and accurate understanding of your customers. This type of analysis can provide you with the data and insight necessary to identify your most profitable customers while delivering communications that they respond to.

To help you optimize your email programs and successfully deliver value to both your customers and your business, Experian CheetahMail has identified the following best practices and proven strategies for campaign testing that every email marketer should employ.


The Control Group

Before running tests, you must determine who is in your control group. A control group is defined as a segment of people who will not be exposed to the marketing elements you’ll be testing.

To garner the most precise test readings, group members should exhibit the natural attributes and average behaviors of your customer base. The ‘control’ or ‘business as usual’ email should contain what you normally send to your customers. For example:

  • Use the same template as previous emails
  • Utilize the exact type and placement of imagery as before
  • Replicate the same headlines and offers


Short-Term Testing

Once control groups have been determined, it is time to execute the test. If you want to gauge how a single element of an email will affect performance, split testing can easily be adapted to examine any aspect of your program, including creative, calls to action, subject lines and offers.

Split tests provide a simple way to help companies make short-term decisions about the value of one implementation versus another. Most marketers create split tests by dividing their entire list into two parts: A and B.

However, when it is not essential that new campaign attributes be tested across your total subscriber list, we recommend running a ‘10-10-80 split’ to maximize response. To do so, first send the two test emails to a statistically significant percentage of your list (for example, 10 percent will receive Version 1 and 10 percent will receive Version 2). Then, after a predetermined time period — generally a few hours to a full day — choose a winner based on the response metrics and deploy the winner to the remaining 80 percent.

Tip: Split tests are a perfect way to address internal debates over copy and imagery.


Long-Term Testing

Businesses will often want to test a strategy over a period of time. This is especially important for email frequency testing, and it’s necessary for any business with significant seasonal variations. Long-term, longitudinal testing is also crucial if campaign influencers, such as promotions and special offers, change frequently or if major business decisions will be based on the results.

The main point to remember when testing over time is that at least one component of the campaign must remain constant. The constant should be either what you are testing (i.e., creative, subject line or offer) or the group of customers within the test group.

Timing can be a crucial factor in understanding campaign results. During the testing period, be mindful of external campaign influencers that can affect results. Create holdout groups for tests where you need to measure incremental lift or distinguish results from other marketing efforts, such as a direct-mail promotion or product sampling.

Tip: For long-term tests, we suggest that you distinctly separate your test groups from the rest of your list. Each test group should represent a statistically significant part of your file to gauge accurate results. The remainder of the file would be the control group.

Listed below are a number of sample scenarios and controls that illustrate how longitudinal splits tests can be executed.


Subject Line Test Scenario: A retailer with a product-centric email program wants to determine if there are significant variances in the response rates garnered by two different subject lines — one that is general and another that is more product-specific.

  • Test group A: [control] subject line 1, “Check out our new bangles, bracelets and bags”
  • Test group B: subject line 2, “Hot new accessories”

Test group A — our control — receives subject line 1, while test group B — an equal-size and evenly assorted cluster of subscribers — receives subject line 2. The group with the highest open rate is indicative of the better-performing subject line.

Tip: We recommend that you consider testing subject lines over a period of time to account for influences such as the offer and the creative. Suggested subject line tests are personalized versus non-personalized, offer versus no offer, short versus long and branded versus non-branded.

Also, pay attention to the searches run on your Web site and the organic searches that bring you traffic from major search engines, and consider using words from the most popular searches in your subject lines.


Subject Line and Creative Test Scenario: Now let’s say another retailer that traditionally includes an offer in the subject line wants to test new subject lines in addition to fresh creative in an effort to increase click rates.

Because two variables are being tested, there would have to be a total of four test groups sorted in the following fashion:

  • Test group A: [control] “Save 15 percent off spring dresses” with creative 1
  • Test group B: [control] “Save 15 percent off spring dresses” with creative 2
  • Test group C: “New, beautifully stylish spring dresses” with creative 1
  • Test group D:  “New, beautifully stylish spring dresses” with creative 2

Tip: Make sure that email creative is consistent for various subject lines. For example, “15 percent off” and “New, beautifully stylish spring dresses” should have the same offer within the creative template, although the creative design itself may be different.


Creative Layout Test Scenario: A media company wants to create a new email template that uses less imagery, reducing the visual complexity of the email and the amount of time dedicated to its creation. To test the response garnered from the simplified layout, one postcard–style image is tested against the control.

  • Test group A:  [control] template with primary, secondary and tertiary areas of content
  • Test group B: postcard–style with one main image

Tip: Two of the most indicative metrics for creative testing are the click-through rate and the click-to-open rate. General open rates are of little to no importance when determining best-performing creative, given that many of your recipients may not be using a preview pane.

Offer Test Scenario: An online grocer wants to test the influence of increasing the minimum spend required to qualify for free delivery.

  • Test group A: minimum spending requirement of $175
  • Test group B: minimum spending requirement of $150
  • Test group C: [control] minimum spending requirement of $100

Tip: The creative look and feel should be the same for each segment. Also, make sure that the subject line is consistent so that open rates are not skewed.


Scheduling Test Scenario: A convenience store chain is trying to determine the optimal time of day to send a weekly circular email. The email is typically sent on Sundays at midnight.

  • Test group A: [control] midnight
  • Test group B: 6:00 a.m.
  • Test group C: 10:30 a.m.
  • Test group D: 3:00 p.m.

Tip: As a best practice, run time-of-day tests over the course of at least one month to eliminate weekly offers from influencing results. If you mail once a week or less, it may be worth extending your time-of-day testing an extra two or three months. On the other hand, for day-of-week testing, it is a best practice to limit the test window to one week, minimizing outside factors.

Multivariate testing: As mentioned in the previous section, split testing allows companies to test the best-performing option inside a single facet of an email campaign. Testing at this capacity is great for businesses looking to make minor analytic-based changes. On the other hand, for those trying to evaluate campaigns on a number of levels, simple split testing is neither practical nor timely.

For example, to launch a new series of emails to recently acquired subscribers, you may want to test attributes similar to the following:













Version 2

Using split testing, you would have to run at least 80 different scenarios. Sample sizes and control groups most likely would be too small to drive meaningful results, and the whole process would be extremely tedious and time-consuming. As a practical solution, try multivariate testing, or the simultaneous testing of a variety of elements.
With multivariate testing, the same 80 or so combinations can be mathematically condensed into just 10 test groups:










It is important to note that a test design this complex must be developed by a trained statistician or a professional third party.



Whatever testing methodology you choose to implement, remember that no company should speak to all of its customers with a single message.

In an environment where marketers must make the most of what they have, acting upon a sophisticated email testing and analytics program will deeply engage your customers and render significant revenue for your business.
By running well-planned, methodological testing strategies on your email campaigns, you’ll be able to drive revenue to new heights and imprint your brand deep into the minds of your customers.

Source: Whitepaper by Experian CheetahMail -