experiments revenue, power and guardrails

Invocation#

echo-cli experiments revenue --experiment <experiment-id>
echo-cli experiments power --app <web or mobile source> --exposure-event <event> --conversion-event <event> --days <n> --min-effect <share> --variants <n> --verdict-sigmas <n> --power-sigmas <n>
echo-cli experiments guardrails --experiment <experiment-id> --by <name>
echo-cli experiments stops --experiment <experiment-id>

Inputs and output#

revenue writes experiment_id and variants (variant_key, exposures, payers, revenue and revenue_per_exposure, each per currency). power writes app_id, since, days, exposure_event, conversion_event, exposed, converted, baseline, people_per_day, min_effect, variants, verdict_sigmas, power_sigmas, sample_per_variant, total_sample and days_needed. guardrails writes readings (variant_key, guardrail, event_name, exposures, hits, share, bound, direction, breached) and the stops it made; stops writes every stop, newest first (id, experiment_id, variant_key, guardrail, share, bound, direction, exposures, hits, stopped_by, stopped_at).

State effect#

Every verb calls Echo Web's authenticated /api/experiments/revenue, /power or /guardrails with ECHO_INTERNAL_BEARER at ECHO_WEB_BASE_URL; the Experiment revenue and guardrails and Experiment sample size sections of the tracking screen, and Echo Desktop's Experiment verdict and Experiment sample size sections, call the same operations, and echo-mcp reads revenue and stops as echo_experiment_revenue and echo_guardrail_stops. revenue credits to a variant every stored web or mobile event of its assigned subjects, after their assignment, with a positive revenue property and a currency, as the billing webhooks store payments, summed per currency and never converted; a payment belongs to a subject when it names the same person, the identifiers the identity edges link joined as identity show joins them, so a purchase RevenueCat stored under the app user id the store knows is credited to the subject the experiment assigned under its web or app id. power reads the source's stored events of the last --days days — a web source's web events or a mobile app's mobile events — : the people (joined across identifiers) who sent --exposure-event, how many then sent --conversion-event (the baseline), and how many arrive a day; the sample per variant is (verdict sigmas + power sigmas)² × (p(1−p) + q(1−q)) ÷ min-effect², p the baseline and q the baseline plus --min-effect; nothing is stored. guardrails measures each guardrail the experiment declares in guardrail_specs (key, event_name, window_hours, bound and direction max or min) for each active variant with at least the experiment's min_exposure_per_variant exposures, as the share of its subjects who sent the event within the window after assignment; a share above a max bound or below a min bound stops the variant: its state becomes quarantined, so assignment no longer returns it, and the stop is stored in adaptive_guardrail_stops with the measured values. Echo Web also evaluates every running experiment that declares guardrails hourly at /api/cron/experiment-guardrails.

Refusals#

• A wrong invocation is refused before anything is sent: "experiments takes <every verb's usage, separated by semicolons>; got <word>", "experiments <verb> does not take <flag>; it takes …", "<flag> needs a value", "experiments <verb> needs <flag>" (so power without --min-effect is refused "experiments power needs --min-effect") and "<flag> must be a number of <unit>; got <value>".
• Echo Web answers a bad request with "experiment_id is required: …", "min_effect is required: the smallest change of the conversion rate, as a share, the experiment must be able to detect", "<field> must be a positive number: …; got <value>", "days must be a whole number of days; got <value>", "variants must be a whole number of at least 2: a test compares a variant with another; got <value>", "the baseline <p> plus min_effect <e> is above 1: no rate can rise that far", "experiment <id> declares no guardrail, so nothing can stop a variant" or "guardrail <n> of experiment <id> needs key, event_name, a direction of max or min, a bound and positive window_hours; got <spec>"; and not found with "no adaptive experiment has id <id>; echo-cli experiments lists them", "experiment <id> has assigned no subject yet, so no payment can be credited to a variant", "no events were found for the source <app> since <instant>; …" or "no one sent <event> on <app> since <instant>, so the surface has no measured traffic". The command prints Echo Web's sentence and exits 1.