A Hard Week: Experiments With Carryover and Interference
A one-week guide to plan, run, and decide on experiments with carryover, interference, and downstream effects.

A hard week plan to answer a ship or not ship decision when your change has carryover, interference, or downstream effects. You will end with a decision rule, an experiment you can run, and an analysis plan that holds up.
- Start from the shipping decision. Define win, lose, and iterate.
- Map causal paths: immediate, carryover, interference, downstream.
- Pick a design that isolates the path you need for the decision.
- Align metrics to the decision window. Pre-specify checks.
- Analyze with ITT for the decision. Explain with supporting cuts.
Always start from the ship/no-ship decision. Then design back.
Who this is for and what you get by Friday
For product managers testing policies or features that shape behavior beyond one session. Pricing, freemium limits, sharing prompts, and network effects live here.
By Friday you will have:
- A clear ship/no-ship decision rule tied to business thresholds.
- A causal map for the change, including carryover and interference.
- A chosen design: holdout cohorts, stepped rollout, switchback, or cluster randomization.
- Primary and secondary metrics with the right time window.
- Pre-specified heterogeneity checks and stopping rules.
- An analysis plan: intent-to-treat (primary), plus per-protocol and mediation for the story.
- Risk bands around outcomes, and next steps if uncertain.
Monday: Start from the shipping decision
State what you will do if the test wins, loses, or is inconclusive. Include rollout speed, guardrails, and who decides.
Define minimum improvement to ship and maximum acceptable downside on key risks. Prefer absolute changes you can defend.
Choose the time horizon that matches the business effect. If habits form, measure long enough to see them.
Tip: Capture this in a Hypothesis card so the team sees the same decision rule.
Monday afternoon: Map causal pathways
List the ways the change can move your outcome:
- Immediate effect: one-session response after exposure.
- Carryover: user learning or habit that persists across sessions.
- Interference: users affect each other (invites, feed crowding, congestion).
- Downstream economics: pricing mix, cannibalization, support load, refunds, or partner effects.
Mark which paths matter for ship/no-ship. Your design will isolate those.
Tuesday: Choose a design that fits the paths
Pick the smallest design that answers the decision. Use the unit and timing that match the pathway.
| Design | Use when | Notes | Risks |
|---|---|---|---|
| Cohort holdout | Carryover, downstream effects over weeks | Randomize by signup or week cohorts. Keep some cohorts unexposed for the full window. | Longer run; risk of contamination if users mix across cohorts. |
| Stepped rollout | Need early read with time trends controlled | Randomize rollout order across units (cohorts/regions). Analyze with difference-in-differences or time fixed effects. | Requires stable trends or modeling of shocks. |
| Switchback | Short carryover at system level (e.g., ranking, pricing) | Alternate on/off by time blocks for the same unit. Choose blocks longer than carryover. | Time-of-day/week confounds if blocks too short or not balanced. |
| Cluster randomization | Interference within groups or networks | Randomize at cluster level (team, classroom, region, shard). Analyze at cluster level or with cluster-robust errors. | Fewer effective units; power depends on intra-cluster correlation. |
Combine designs as needed. Example: stepped rollout across weekly signup cohorts, with some cohorts held out as long-term controls.
Wednesday: Metrics, checks, and stopping rules
- Primary metric matches the decision window and unit.
- Secondary metrics explain why: exposure, adoption, invites, ARPU, retention.
- Heterogeneity to pre-specify: new vs existing users, cohorts, clusters.
- Stopping rules for safety and for futility. Avoid peeking for wins outside the plan.
- Exposure logging and compliance definition upfront.
- Data quality: missingness, contamination, clock drift, and id resets.
Align metrics to pathways:
- Immediate: session conversion, click-through, revenue per exposure.
- Carryover: repeat use over weeks, habit formation markers.
- Interference: per-cluster outcomes, invite acceptance, network saturation.
- Downstream: paid conversion mix, churn, support tickets.
Pre-register windows and definitions so the result maps to ship/no-ship without debate.
Thursday: Launch, monitor, and protect the estimate
- Randomization integrity: verify balance on pre-period metrics at the chosen unit.
- Guardrails: pause rules for critical metrics. Define them before launch.
- Exposure and compliance: log first exposure time and total exposures. Define compliant users for per-protocol analysis.
- Interference hygiene: minimize cross-treatment contamination. Keep clusters intact; avoid cross-invites across arms if that matters for the path.
- Time controls: for stepped or switchback, record block and calendar effects.
Friday: Analyze, report risk, decide
Primary: intent-to-treat (ITT). Estimate the effect of assignment on outcomes over the decision window. This supports ship/no-ship because rollout follows assignment, not perfect compliance.
Explain: add per-protocol (compliant users) to bound effects if compliance is low, and mediation to attribute through adoption, invites, or price mix. Keep these as supporting, not decision-making, unless pre-specified.
Risk bands and thresholds: show uncertainty around the effect. Compare the lower and upper bands to your practical thresholds. Decide:
- Ship if the lower band is above the ship threshold and guardrails are met.
- Do not ship if the upper band is below the threshold or any guardrail is breached.
- Iterate or extend if the band straddles the threshold and value of information is positive.
State the decision, rationale, and next step in one page.
Common mistakes
- Measuring the wrong window. Shipping based on a short-term bump that fades or reverses.
- Ignoring interference. Treating a social or network product as if users are independent.
- Stopping on transient peaks. Peeking early and calling it. Respect the plan.
- No pre-defined decision. Debating the goalpost after you see the data.
Worked example: Freemium limit with carryover and network effects
Context. You change a freemium limit. It may shift usage, purchases, and invites. Effects may carry over as users learn the new limit. Invites create interference. Downstream economics show up in trial conversions and retention over weeks.
Design. One-week stepped rollout across weekly signup cohorts, plus an 8-week cohort holdout:
- Units: weekly signup cohorts. Some cohorts are held out for the full 8 weeks (never see the change). Other cohorts roll in during one week, in randomized order.
- Interference control: keep invite flows within cohorts where possible (e.g., invite suggestions scoped to cohort or batch rollout by clusters that align to social graphs). If that is not possible, upgrade to cluster randomization aligned to social clusters.
- Carryover coverage: the 8-week window captures habit changes and subscription decisions.
- Time control: stepped rollout lets you separate calendar shocks from treatment effects.
Primary and secondary metrics.
- Primary: trial conversion over 8 weeks from cohort start.
- Secondary: invite rate per user, invite acceptance, week-8 retention, paid retention if applicable, support contact rate.
- Guardrails: day-7 churn, refund rate, critical error rates.
Sample sizing. Size by cohort. For the primary metric, choose a minimum detectable effect and power. Use baseline conversion for recent cohorts to compute the number of users needed per cohort and the number of cohorts to include. Adjust for clustering if you randomize by cluster: inflate by the design effect using an estimate of intra-cluster correlation. For stepped rollout, ensure enough cohorts per step to estimate time effects.
Pre-specification. Before launch, write:
- Decision rule: Ship to 100% if the ITT effect on 8-week trial conversion exceeds the practical threshold and guardrails are met. Iterate if the confidence band straddles the threshold and the shortfall on conversion is within the team’s acceptable risk. Do not ship if the upper band is below the threshold or guardrails fail.
- Heterogeneity checks: new vs returning signups, high vs low inviters, regions.
- Stopping: no early stopping for wins. Pause only on pre-defined safety breaches.
Implementation notes. Randomize rollout order of eligible cohorts at the start of the week. Keep holdout cohorts fully unexposed for 8 weeks. Log first exposure timestamp, total exposures, invites sent and accepted, trials started, conversions, and churn. Tag every event with cohort and step.
Analysis.
- Primary (ITT): compare treated vs holdout cohorts on 8-week trial conversion. For stepped cohorts, include time fixed effects to absorb calendar shocks.
- Per-protocol: among exposed users who hit the limit at least once, re-estimate to bound the effect size. Keep as explanatory.
- Mediation: decompose via invite activity and purchase intent proxies to explain the path.
- Interference check: compare cross-cohort invite rates. If spillovers are material, interpret effects as policy-on-network rather than user-level effects.
- Risk bands: present uncertainty for the primary effect and for downside guardrails. Make the ship/no-ship call per the written rule.
Decision. If the 8-week ITT lift on trial conversion clears the threshold and guardrails hold, ship to 100% after the week, continue monitoring the holdout cohorts for the full 8 weeks to confirm stability. If not, iterate on the limit or pricing, and plan a follow-up with the same decision framing.


