“An A/B test is one prompt away” is a useful ambition. It is also the kind of sentence which can conceal a substantial platform build-out and a statistically indefensible product decision.
The prompt is not the experiment.
A prompt can express a hypothesis, propose variants, generate implementation scaffolding and perhaps configure a safe preview. The platform still has to establish who is eligible, what receives the treatment, whether assignment and exposure are trustworthy, which measures answer the question, when the test may stop, what protects the product while it runs, who interprets the result and how the change is rolled back.
When those things are integrated, AI really can compress the product-learning loop. Without them, it mostly compresses the time between an idea and an ambiguous production change.
Start with an experiment contract
For this series, an experiment contract is the versioned record agreed before rollout which defines the hypothesis, eligible population, assignment and allocation design, intervention, measures and instrumentation, estimand and analysis design, decision and stopping rules, rollback boundary, and accountable owners. This is a demming.dev working definition, not an industry standard.
The contract is deliberately broader than a statistical analysis plan. It joins a product decision to the software and data paths which must make that decision trustworthy.
| Contract field | Question it must settle | Typical owner |
|---|---|---|
| Hypothesis | Which behavior should change, why, and in which direction? | Product owner |
| Population and eligibility | Which pre-treatment rule defines who or what may enter, and who must not? | Product, data, privacy |
| Assignment and allocation | What is randomized, by which mechanism and ratio, with which stratification and persistence? | Experimentation and engineering |
| Intervention and window | What differs between control and treatment, which version is fixed, and over which exposure and outcome window? | Product, design, engineering |
| Measures | Which success measure answers the question, and which guardrails and diagnostics constrain interpretation or rollout? | Product, data, engineering |
| Instrumentation | Which assignment, exposure and outcome events are required, with which schemas and identifiers? | Data and engineering |
| Estimand and analysis | Which effect, population and time window are being estimated; at which analysis unit; under which clustering or interference assumptions, estimator and uncertainty calculation? | Experimentation specialist |
| Decision and stopping | Which Type-I error budget, power and MDE, baseline assumptions, horizon or maximum information, sequential rule and multiplicity family apply? | Experimentation specialist and product owner |
| Rollout and rollback | What may progress, in which increments, and which condition reverses the intervention? | Production owner |
| Decision record | Who calls ship, revise, continue or stop, and where is the evidence retained? | Accountable product owner |
An agent may help fill this in. It should not be able to silently omit the awkward fields.
The loop is one connected system
The governed loop is straightforward to draw and surprisingly difficult to make real:
- A product capability is inventoried and declared experimentable.
- A hypothesis and experiment contract are reviewed.
- An agent proposes a bounded implementation and flag configuration.
- CI, evaluation and environment checks produce acceptance evidence.
- Assignment and exposure begin inside the authorized population and rollout limit.
- Telemetry flows through governed data contracts into reproducible analysis.
- Statistical and product interpretation produce a recorded decision.
- The platform promotes, revises, continues or rolls back within the predeclared boundary.
Feature flags are useful in that loop, but they are not the loop. OpenFeature’s Flag Evaluation API is independent of a flag control plane or vendor; the specification separately defines providers, evaluation context including targeting data, and hooks around evaluation. That is a sound interoperability layer. It does not establish that an assignment is random, that an exposure was observed, that a metric is causal, or that a rollout is authorized.
The same distinction applies to AI. Generating the flag, code and dashboard faster is valuable. None of those artifacts grants statistical validity or production authority.
Assignment is a systems decision
“Fifty percent control, fifty percent treatment” is incomplete until the assignment unit is explicit.
If treatment is assigned per session but users return across sessions, a person may see both variants. If it is assigned per account while the analysis treats users as independent, observations inside an account may be correlated. I would treat any plausible spillover between units as its own design decision rather than letting a platform default assume it away. The correct design depends on the product; there is no universal unit or estimator hiding in a platform default.
Research on online experiments has shown how more complex randomisation mechanisms can invalidate naive independence assumptions and underestimate variance. The narrow point is not that one paper solves every clustering or interference problem. It is that the assignment mechanism and the analysis model must agree. (“Trustworthy analysis of online A/B tests: Pitfalls, challenges and solutions,” WSDM 2017)
This is where product architecture enters what looks like a statistics problem. Stable assignment needs a stable identity at the chosen unit. Targeting needs an eligibility decision based on data available at the right time. A user moving between devices, an account changing tier, cached flag state, asynchronous propagation or a late-arriving entitlement update can all change what “assigned to treatment” means in practice.
The platform should make the assignment contract visible and testable. It should also make unsupported designs hard to launch.
Exposure telemetry belongs to the causal design
Assignment says what should have happened. Exposure says what was actually presented.
That difference matters. A person can be assigned to a feature and never reach it. A client can fail before rendering it. An event can be dropped, duplicated, delayed or joined to the wrong unit. If the outcome population is filtered differently from the assigned population, the final dashboard may look precise while answering a different question.
A defensible default is to preserve the randomized comparison and estimate an intention-to-treat effect over the assigned eligible population. Filtering to people recorded as exposed can condition on a post-assignment event affected by treatment or telemetry and destroy that protection. Missing exposure is therefore an integrity diagnostic, not permission to discard the unexposed units.
Triggered or exposure-based analysis can still be useful, particularly when only a small part of the assigned population could encounter the intervention. The estimand and trigger have to be declared in advance, the trigger has to represent who would have been affected in either arm, and the implementation may need counterfactual logging, its own SRM check and a complement analysis. Per-protocol or treatment-on-treated estimates answer yet another question and need their own assumptions. (Microsoft Research, post-experiment triggered-analysis guidance)
Telemetry loss does more than reduce the row count. Depending on where and why it occurs, it can reduce power and introduce bias. Work on trustworthy experimentation under telemetry loss treats the data path itself as part of experiment validity, which is exactly the platform lesson: the assignment, exposure and outcome pipelines require explicit integrity checks. (“Trustworthy Experimentation under Telemetry Loss,” CIKM 2018)
One particularly useful diagnostic is sample-ratio mismatch (SRM): the observed allocation differs from what the assignment design predicts beyond plausible random variation. SRM is a symptom, not a root cause. It points to a defect somewhere in assignment, eligibility, data processing or telemetry; the paper’s taxonomy goes considerably deeper. Ignoring it can reverse a product decision; passing it does not prove everything else is sound. (“Diagnosing Sample Ratio Mismatch in Online Controlled Experiments,” KDD 2019)
I would therefore treat missing exposure data and SRM as release-quality signals, not as footnotes an analyst discovers after the test.
Decide what can be detected before looking at the result
An experiment which cannot plausibly detect a decision-relevant change is not made useful by waiting for a green number.
The contract needs the effect size which matters to the product and a design with adequate power to detect it under its assumptions. Microsoft’s experimentation guidance describes statistical power as the probability of detecting an effect when the alternative holds, and the minimum detectable effect (MDE) as the smallest change the design can detect with a selected high probability. Exact calculations depend on the measure, variance, assignment, estimator and decision design; the platform should not hard-code one sample-size formula as universal truth. The contract should record the experiment-specific error budget, target power, baseline and variance assumptions, planned horizon or maximum information, cluster inflation where relevant, sequential method and multiplicity family. (Microsoft Research, STEDII properties of a good metric)
This forces a useful business conversation. Is the smallest economically meaningful improvement detectable in a reasonable period? If not, should the intervention be larger, the measure become more sensitive, variance be reduced, more traffic be eligible, or the decision be made using different evidence? “Run it anyway” is sometimes a valid exploratory choice, but it should be labeled as such.
Peeking and multiplicity need declared rules
Product teams will look at a running experiment. Pretending otherwise is not governance.
The error is allowing every glance to become an undeclared stopping rule while interpreting conventional fixed-horizon p-values as if the sample size had been fixed in advance. Repeatedly stopping when the result first looks favorable changes the error properties. Sequential methods, including always-valid inference, can support continuous monitoring under defined procedures; they are not a license to improvise the procedure after seeing the data. (Johari, Pekelis and Walsh, “Always Valid Inference,” arXiv:1512.04922)
Multiplicity appears elsewhere too: several treatments, many success measures, many segments, and repeated looks each create more opportunities to mistake noise for a discovery. The right correction depends on the decision structure. A useful platform records that structure, applies the selected method consistently and warns when the analysis no longer matches the contract. (Microsoft Research on during-experiment patterns, experiment alerting)
Time adds another complication. Novelty can make an intervention look unusually good or bad at first. Weekends, campaigns, releases and seasonal events can change the mix of behavior. Segmenting by date can expose those patterns, but a chart by day does not itself remove bias or prove that the result will generalize. It is a diagnostic feeding a human interpretation.
A significant measure is still only evidence
Even a properly randomized experiment can be misread.
A primary measure may improve while a latency, reliability, retention or complaint guardrail deteriorates. A ratio measure can move because its denominator changed. A click measure can be sensitive but irrelevant to the intended outcome. A segment result can be an unstable post-hoc story. Research cataloguing recurring metric-interpretation failures is a useful antidote to dashboard certainty. (“A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls,” KDD 2017)
The platform should make three classes of measure explicit:
- success measures, tied to the product hypothesis;
- guardrails, capable of stopping or constraining rollout when harm or system degradation appears; and
- diagnostics, such as assignment balance, exposure completeness and event freshness, which help determine whether the result is interpretable, qualified or invalid.
The final decision still needs judgment. Statistical evidence informs ship, revise, continue or stop; it does not own that verb.
An illustrative data and BI path
The analysis layer does not require one blessed vendor stack. One coherent composition stores exposure and outcome evidence in open Parquet files on S3, lets shared query engines such as Athena or Trino read the same data, uses DuckDB where a compact reproducible analysis is useful, and presents decision-facing results through Metabase. The architectural point is shared object-storage evidence, interoperable query paths, local reproducibility where appropriate, and a BI surface close to the product decision. AWS documents Athena querying S3 and Parquet; Trino and DuckDB document Parquet access from object storage; Metabase documents an Athena connection. Those are compatibility facts, not a recommendation for every workload. (AWS Athena, AWS Athena data and Parquet, Trino S3 support, Trino file formats, DuckDB Parquet, DuckDB S3 import, Metabase Athena connection)
Treat this as one reference composition, not a prescribed or singular deployed topology. It says nothing by itself about catalogs, privacy, cost, concurrency, retention, freshness or access control, and the inherited-system pattern below is a separate architectural concern.
What matters is the integration contract: assignment and exposure identifiers survive the path; schemas are versioned; late and duplicate events are handled; metric definitions are reproducible; access follows policy; and the decision record can point back to the exact data and experiment version it used.
Experimentability raises the systems bar
A broad experiment capability is not a thin feature-flag UI. It puts pressure on the entire product platform.
Every capability declared experimentable needs a bounded way to vary behavior; the inventory should also record capabilities excluded by risk, ethics, law, reversibility or insufficient volume. Operational schemas need to represent substitutes without a forest of one-off columns. Events need stable meaning. Eventually consistent integrations need an explicit view of when eligibility, assignment and exposure become authoritative. Tight coupling becomes visible when a harmless-looking variant requires coordinated deployment across several inherited services.
That is useful pressure. It turns “legacy modernization” from an abstract program into a sequence of product-learning constraints. Decoupling a service, repairing an event contract or making an assignment key stable is no longer just architectural tidiness; it enables a safer class of experiments.
The work runs in parallel across product, operational systems, data and delivery. AI helps with the cruft—inventory, scaffolding, contract generation, test creation, analysis preparation. It does not make distributed state, causal inference or rollout risk disappear.
The target experience can still feel close to one prompt away to a product manager. That simplicity is earned elsewhere: in a contract with enough decisions to analyze, delivery and data paths which preserve it, and an owner who can act on the evidence or stop.
Before production authority increases, the product owner should be able to inspect the eligible population and assignment unit, exposure completeness, estimand and uncertainty, success and guardrail results, stopping rule, and rollback condition. If the platform cannot produce that record, the variant stays in test or staging. That is the minimum evidence boundary for turning faster change into product learning.