A/B testing
Split an automation step across template versions and promote the winner.
A/B testing happens at the template-version level, on an automation step. You publish two (or more) versions of a template, attach them to a step with weights, and let the platform assign contacts deterministically.
Publish variant versions
Create variant versions of a template (a different subject or body) and publish each. Published versions are immutable, so a variant is a stable thing to test against.
Attach the split
Attach the versions to a send step with weights (email_set_ab_test). Assignment
is deterministic per contact-run — a contact always sees the same variant,
and the split is reproducible.
Read the results
email_get_automation_metrics returns per-variant delivered / open / click
buckets plus an abWinners readout. It only calls a leader once each arm has
enough delivered volume and the gap is material — otherwise it says "keep the
test running", so you don't chase noise.
Promote the winner
When a leader is called and you decide to ship it, email_promote_ab_winner
collapses the split to the winning variant so every contact gets it. It
re-verifies the winner at apply time and refuses if the arms no longer clear the
thresholds — a stale reading can't force a bad promotion. It's reversible.
email_get_optimization_plan surfaces "winner ready to promote" as a ranked
action with the exact apply call, so you don't have to hunt for it.
Minimum sample before trusting a result: roughly 200 delivered per variant for open rate, ~1000 for click rate. A variant that wins opens but loses clicks is a clickbait subject — judge on the downstream metric.