experiments

Invocation#

echo-cli experiments
echo-cli experiments outcomes --experiment <experiment-id> --sigmas <n>
echo-cli experiments verdict --experiment <experiment-id> --sigmas <n>

Inputs and output#

With no verb, writes experiments and variants arrays with status, weights, exposures, rewards, and timestamps; assignment subject identifiers and private variant config are not selected. outcomes and verdict write experiment_id, reward_event, window_hours, sigmas, variants (variant_key, exposures, conversions, rate), comparisons (leader, other, difference, low, high) and winner (null while any interval reaches 0).

State effect#

With no verb the command reads every row of adaptive_experiments and adaptive_variants. outcomes and verdict call Echo Web's authenticated /api/experiments/verdict with ECHO_INTERNAL_BEARER at ECHO_WEB_BASE_URL; the Experiment verdict section of the tracking screen and of Echo Desktop's GEO screen call the same operation, and echo-mcp reads it as echo_experiment_outcomes. A variant's exposures are the subjects assigned it in adaptive_assignments; its conversions are those of them whose web or mobile events, by user id or anonymous id, include the experiment's reward_spec event_name within its window_hours after their assignment. Between the variant with the highest rate and every other, the interval is the difference of rates ± --sigmas standard errors of that difference. outcomes answers the verdict as it stands; verdict answers only when every interval lies above 0, naming the winner, and is refused while any still reaches 0. Nothing assigns a subject, alters a weight or graduates a variant.

Refusals#

• A wrong invocation is refused before anything is sent: "experiments takes <every verb's usage, separated by semicolons>; got <word>", "experiments <verb> does not take <flag>; it takes --experiment, --sigmas", "<flag> needs a value", "experiments <verb> needs <flag>" and "--sigmas must be a number of standard errors; got <value>".
• Echo Web answers a bad request with "experiment_id is required: the adaptive experiment to judge", "sigmas is required: how many standard errors each interval spans", "sigmas must be a positive number: how many standard errors each interval spans", "experiment <id> declares no reward event in its reward_spec, so no conversion can be counted" or "experiment <id> declares no positive reward window_hours in its reward_spec, so a conversion has no window"; not found with "no adaptive experiment has id <id>; echo-cli experiments lists them", "experiment <id> has assigned no subject yet, so there is nothing to compare" or "experiment <id> has assigned only variant <key>, so there is nothing to compare it with"; and verdict, while an interval reaches 0, a conflict with "no winner yet at <n> standard errors: <leader> − <other> is <difference> with interval [<low>, <high>]; …, which still reaches 0". The command prints Echo Web's sentence and exits 1.