Email A/B testing sends two or more versions of one email to random slices of a list and keeps the version that wins on a metric chosen in advance. Test one variable at a time, judge it on clicks or revenue, fix the sample size before sending, and keep a holdout group to measure real lift.
Every major email platform ships an A/B button, so the mechanics are easy. The judgment is harder. Teams test subject lines on open rate, call the result after an afternoon, and roll the winner into every future send. A quarter later nothing has moved. The sections below walk through the decisions that separate real email A/B testing from a coin flip, starting with what you test and ending with the calendar problem the holiday season creates.
Email A/B testing starts with one variable and a written hypothesis
An email A/B test can only teach you something if a single element changes between versions. Mailchimp's tool makes that the rule: an email test uses one of four variables (subject line, From name, content or send time), allows up to three variations, and sends each contact only one combination, chosen at random (Mailchimp). HubSpot's version, available on Marketing Hub Professional and Enterprise, splits a chosen percentage of recipients between versions A and B, picks a winner on open rate, click rate or click-through rate after a test length you set in hours, and sends the winner to everyone else (HubSpot).
Good email A/B testing starts on paper: write the hypothesis down before you build the variant. "A product-led subject line will raise click rate among repeat buyers" gives you a variable, a metric and an audience; "try an emoji" gives you none of those. HubSpot also asks you to choose a fallback version for inconclusive results, which is a useful discipline on any platform: decide in advance what ships if the test proves nothing. The hypothesis should also name the segment, since a result from your most engaged buyers rarely transfers to the whole file. Our guide to email list segmentation covers how to define those groups so the test audience stays stable from one send to the next.
Open rate is a weak winning metric for email A/B testing
Most subject line tests still pick a winner on opens, and Apple's privacy feature has made that number unreliable. Apple says Protect Mail Activity downloads remote content in the background by default, "regardless of whether you engage with the email" (Apple). Klaviyo's help center spells out the consequence for testing: Apple's machine opens are included when an A/B test uses open rate as its winning metric, statistical significance does not establish that anyone intentionally opened the email, and Klaviyo suggests click rate when those opens make results hard to read (Klaviyo).
Pick the metric closest to the outcome you care about and that the test can still measure in its window. For subject lines and From names, click rate is the practical choice. For content and offer tests, use click rate or revenue per recipient, and Mailchimp can select a winner on total revenue when a store is connected (Mailchimp). We explain why opens slipped from the scorecard in email marketing metrics that still matter, and Apple's latest changes are covered in our post on iOS 27 email marketing.
Email A/B test sample size decides which lifts you can detect
Sample size is the constraint most email A/B testing advice skips, and it decides whether a test can find anything. Customer.io's documentation says testers typically aim for 95 percent confidence and offers 500 recipients per variation only as a rule of thumb, urging teams to calculate sizes for their own needs (Customer.io). HubSpot recommends sending an A/B tested email to at least 1,000 contacts (HubSpot). Both floors are far below what a low click rate demands.
Run the numbers before any email A/B test goes out. Using the standard two-proportion formula at 95 percent confidence and 80 percent power, our own calculation shows that a list with a 2 percent click rate needs about 21,000 recipients per variant to detect a 20 percent relative lift (2.0 to 2.4 percent), about 81,000 per variant for a 10 percent lift, and about 3,800 per variant for a 50 percent lift. Evan Miller's free sample size calculator uses a slightly different method and returns somewhat lower figures for the same inputs (about 19,800, 78,000 and 3,300), so use it to check your own baseline; either way, the order of magnitude is the point. If your list cannot reach the number, test bigger changes, such as a different offer or a different email, since small copy tweaks need audiences most senders do not have.
Email A/B testing statistical significance breaks when you peek
Checking a live test and stopping it the moment it looks significant inflates false winners, even when the dashboard says 95 percent. Evan Miller showed that in a worst case, with a significance check after every observation, an experiment where the new version does nothing produces a "significant" result 26.1 percent of the time, and that peeking ten times turns a reported 1 percent significance level into an actual 5 percent (Evan Miller). His fix is simple: decide the sample size in advance and do not trust the result until the test has run its course.
Platform labels help with email A/B testing, provided you read their thresholds. Klaviyo marks a campaign test statistically significant when each variation has more than 50 successful email deliveries and the win probability exceeds 90 percent; a "promising" result (win probability between 75 and 90 percent) should be repeated, and a factor that keeps testing not significant after two or three retests should be dropped for a new one (Klaviyo). Treat 50 deliveries as the point where a label appears, well short of the sample you need. Automated flows are the best place to repeat a test, since a welcome email series keeps sending to new people every day.
An email holdout group measures what A/B tests cannot
An A/B test tells you which version wins; a holdout group tells you whether sending the email earns anything at all. Braze's Global Control Group lets a team set a percentage of all users who receive no campaigns or Canvases, so their behavior can be compared with users who do get messages and the uplift measured over time (Braze). The same idea works in any platform with a random split: suppress a small, random slice from a flow and compare purchases, renewals or churn across both groups.
Holdouts answer the budget questions that email A/B testing cannot. Does the post-purchase series drive repeat orders, or would those customers have come back anyway? Braze advises shuffling control group membership no more than once a month (Braze), which gives behavior time to show up. Our post-purchase email sequence guide measures repeat purchase rate against a holdout for exactly this reason. Keep holdouts away from transactional mail such as receipts and password resets, which customers need whatever the test design.
AI marketing agents now watch email A/B testing results
AI agents inside email platforms can now monitor tests on a schedule, which makes the stopping rule more important. On October 13, 2026, Klaviyo made Composer, its AI marketing agent, available to every customer, with scheduled tasks that run around the clock, reusable "Skills" that encode a team's process, and benchmarks drawn from 205,000 customers (Klaviyo). Klaviyo's own example prompt asks Composer to flag any email whose revenue per recipient is 25 percent below its 90-day average.
An agent that checks results every morning is peeking by design. If it is allowed to act on what it sees, it will declare winners early for the reasons covered above. Write the test rules into whatever process the agent follows: the hypothesis, the winning metric, the fixed sample size and the date the result may be read. Let the agent surface the readout and keep the decision with a person until it has earned more room. Our framework for AI agent governance covers how to set those permissions before an agent can change live flows.
Holiday email A/B testing needs a freeze date
Finish your email A/B testing before November, then ship what you learned. The National Retail Federation forecast on October 15 that November and December retail sales will grow 4.2 percent to $1.013 trillion, with the season defined as November 1 through December 31 (NRF). Holiday traffic is valuable precisely because it is unusual, and Klaviyo advises avoiding factors that can sway an audience, such as retesting near a holiday weekend (Klaviyo).
Use the next two weeks to settle the questions that matter for peak season, such as offer framing, From name and send time, and treat any result from Black Friday week as a seasonal reading and keep it out of January's baseline. Our Black Friday email strategy lays out the send calendar those results should feed. If you want help designing an email testing program, with sample sizes, holdouts and agent guardrails set before the first send, talk to our team.
