A/B Testing for Email Campaigns

A/B Testing for Email Campaigns

A/B Testing for Email Campaigns

A/B testing allows you to send two variations of a campaign (A and B) to a subset of your audience, evaluate which performs better based on your chosen metric, and automatically send the winning variation to the rest of your subscribers. This feature helps you optimize subject lines and email content to maximize your open and click rates.

How it Works

The setup process gives you granular control over your testing variables and audience splits before automatically handling the evaluation. To create an A/B Test, navigate to Campaigns → A/B Tests and click Create A/B Test.

       

  1. Configure Variations: You can create exactly two variations (Variation A and Variation B) to test either your subject line or your email content body, focusing on one variable at a time.
  2. Audience Distribution: You have full control over the percentage split. For example, you can send Variation A to 25% of your audience, Variation B to another 25%, and reserve the remaining 50% for the winning campaign. The audience state is safely "frozen" at the time of launch so the splits remain consistent.
  3. Winning Metric: Choose how success is measured. You can choose between the open rate or click rate.
  4. Limit Values (Thresholds): You can optionally set a target threshold percentage for your metric. This represents the minimum performance level a variation must reach to be declared the winner.

Automatic Winner Evaluation

Once the test is launched, the system operates in the background to monitor performance for your specified test duration (in hours).

Declaring a Winner: If a variation reaches your optional limit value, it wins immediately. If both reach it, the higher rate wins. If the test duration elapses without hitting a limit, the variation with the highest metric value, even by a fraction of a percentage, is automatically declared the winner. In the rare event of an exact tie, Variation A wins by default.

Automatic Sending: Once a winner is declared, the system handles the rest. It automatically creates a clone of the winning campaign and sends it to the remaining portion of your audience.

Important Considerations

Please note the following constraints when configuring your A/B tests to ensure smooth delivery:

Campaigns Only: A/B testing is exclusively available for regular email campaigns. It does not apply to automated workflows or SMS messages.

No Minimum Audience: There is no minimum audience size enforced to run a test.

Test Finality: Variations can only be edited while the overall test is in a DRAFT status. Once a test status changes to RUNNING, it cannot be edited anymore.