compare
Compare a shipped change and return a keep, revert, or extend verdict. Compare is a free read. Each successful call counts as one metered read. Pass ship_date plus window_days, or explicit baseline and compare windows. window_days is an integer from 1 to 90 and a multiple of 7 (default 7, so the largest ship window is 84). The ship day is in neither window: baseline is ship-N through ship-1, and compare is ship+1 through ship+N. Run a day-7 read on ship day + 8 so the last day is complete. If either window includes today UTC, the verdict is extend with reason incomplete_current_day, and that reason overrides every other verdict. goal_event is required and is counted as unique converters (count_mode unique_users): a user who fires the goal more than once counts once. denominator_event defaults to page_view and is counted as unique users. rate = goal_converters / denominator_unique_users. Optional filters use the query filter shape and apply as they do today. A filter on event is rejected. Optional denominator_filters use the same {field, op, value} shape and operators and apply only to denominator events when computing denominator_unique_users. goal_converters still counts only goal users inside that filtered denominator. Omit denominator_filters or pass [] for today's behavior. Max 10 entries. To exclude one-hit 404 page views from the denominator: denominator_filters [{"field":"properties.title","op":"neq","value":"Page not found | grokbotpages.com"}]. rule thresholds are relative fractions (0.15 means +15%). The absolute value of keep_threshold and revert_threshold must be at most 5. Omitted rule fields use the documented day-7 examples (keep_threshold 0.15, revert_threshold -0.1, min_denominator 200, min_goal_events 20 unique converters across both windows). Those examples are not product policy. Verdict order: extend incomplete_current_day; extend below_min_sample; extend zero_denominator; extend zero_baseline when lift is null; revert revert_threshold_met only when lift is at or below revert_threshold and the 90% interval lies entirely below zero; extend inconclusive when the drop is within noise; keep keep_threshold_met when lift is at or above keep_threshold and interval.low is above 0; extend interval_includes_zero; extend no_evidence_of_gain. A drop by itself is not a revert. Each window reports date_from, date_to, day_count, denominator_unique_users, goal_unique_users, goal_converters, goal_users_outside_denominator, count_mode, and rate. The response also includes count_mode, reason_detail, lift, echoed denominator_filters, and a 90% interval (level, method newcombe_wilson_90, estimate rate_difference, low, high, excludes_zero, note). Optional metric compares a numeric property. metric is sum(<property>), avg(<property>), p50(<property>), p75(<property>), or p95(<property>), for example sum(revenue) or p75(cwv_lcp). The property uses the query property allowlist ^[A-Za-z0-9_]+$ with a maximum of 128 characters. Example: metric p75(cwv_lcp), metric_event page_view, better lower, while goal_event stays signup. On hosted, cwv_* vitals are properties on page_view. Prefer the web_vitals tool for the p75 headline, ratings, and shares. metric_event is the event whose property is measured and defaults to goal_event. Set it when the measured event is not the goal. better is higher (default) or lower and is not inferred from the property name. Use better=lower for cwv_lcp and the other Web Vitals. Shared filters apply to the metric event. denominator_filters still apply only to the denominator. sum() compares the sum divided by unique denominator users (per_user true). avg and percentiles compare the aggregate itself (per_user false). Percentiles are nearest-rank, so percentile_method is nearest_rank. With metric set, interval is null and interval_method is none. There is no bootstrap interval in v1. lift is (compare - baseline) / baseline, negated when better is lower, so the existing thresholds keep their meaning. raw_lift is the unnegated ratio. min_denominator is still unique denominator users per window. When metric is set, min_goal_events is numeric values (n) on metric_event across both windows, not unique converters. below_min_sample is extend when n across both windows is under min_goal_events (default 20) or either window is under min_denominator (default 200). incomplete_current_day, below_min_sample, zero_denominator, and zero_baseline stay in that order. A missing compare value, or a baseline of 0 or null, is zero_baseline with lift null. When lift meets keep_threshold or revert_threshold, the verdict is extend with reason no_interval_for_metric, and lift is still reported. Lift between the thresholds is extend / no_evidence_of_gain. The response metric object includes metric, fn, property, metric_event, better, per_user, and percentile_method. Each window still includes the conversion rate and user counts, plus value, n, and skipped. Omit metric and the compare response is unchanged, including the Newcombe interval. Compare stays a free read when metric is set.
Schema valid · SHA-256 1e776c33f7f7…
View input schema
{
"type": "object",
"properties": {
"project": {
"type": "string",
"description": "Project name or ID"
},
"goal_event": {
"type": "string",
"maxLength": 256,
"description": "Goal event counted as unique converters (count_mode unique_users). A user who fires it more than once counts once"
},
"denominator_event": {
"description": "Denominator event counted as unique users. Defaults to page_view. rate = goal_converters / denominator_unique_users",
"type": "string",
"maxLength": 256
},
"filters": {
"description": "Optional filters, same shape as query. Do not filter on event",
"type": "array",
"items": {
"type": "object",
"properties": {
"field": {
"type": "string",
"description": "Field to filter on (user_id, date, country, session_id, timestamp, or properties.*). event is rejected"
},
"op": {
"type": "string",
"enum": [
"eq",
"neq",
"gt",
"lt",
"gte",
"lte",
"contains"
],
"description": "Filter operator"
},
"value": {
"description": "Value to compare against",
"type": [
"string",
"number"
]
}
},
"required": [
"field",
"op",
"value"
]
}
},
"denominator_filters": {
"description": "Filters applied only to denominator events when counting denominator_unique_users. Same shape and operators as filters. Max 10. Omit or pass [] for today's behavior. Example: {\"field\":\"properties.title\",\"op\":\"neq\",\"value\":\"Page not found | grokbotpages.com\"}",
"maxItems": 10,
"type": "array",
"items": {
"type": "object",
"properties": {
"field": {
"type": "string",
"description": "Field to filter on (user_id, date, country, session_id, timestamp, or properties.*). event is rejected"
},
"op": {
"type": "string",
"enum": [
"eq",
"neq",
"gt",
"lt",
"gte",
"lte",
"contains"
],
"description": "Filter operator"
},
"value": {
"description": "Value to compare against",
"type": [
"string",
"number"
]
}
},
"required": [
"field",
"op",
"value"
]
}
},
"ship_date": {
"description": "Ship date (YYYY-MM-DD, UTC). The ship day is in neither window",
"type": "string"
},
"window_days": {
"description": "Days in each ship_date window. Integer from 1 to 90 and a multiple of 7. Defaults to 7. Omit when passing baseline and compare",
"type": "integer",
"minimum": 1,
"maximum": 90,
"multipleOf": 7
},
"baseline": {
"description": "Explicit baseline window. Pass with compare, and omit ship_date and window_days",
"type": "object",
"properties": {
"date_from": {
"type": "string",
"description": "Inclusive start date (YYYY-MM-DD)"
},
"date_to": {
"type": "string",
"description": "Inclusive end date (YYYY-MM-DD)"
}
},
"required": [
"date_from",
"date_to"
]
},
"compare": {
"description": "Explicit compare window. Pass with baseline, and omit ship_date and window_days",
"type": "object",
"properties": {
"date_from": {
"type": "string",
"description": "Inclusive start date (YYYY-MM-DD)"
},
"date_to": {
"type": "string",
"description": "Inclusive end date (YYYY-MM-DD)"
}
},
"required": [
"date_from",
"date_to"
]
},
"rule": {
"description": "Decision rule. Omitted fields use the documented day-7 examples, not product policy. Threshold absolute values above 5 are rejected",
"type": "object",
"properties": {
"keep_threshold": {
"description": "Relative lift fraction required to keep, and only when interval.low is above 0. 0.15 means +15%. Absolute value must be at most 5",
"type": "number",
"minimum": -5,
"maximum": 5
},
"revert_threshold": {
"description": "Relative lift fraction. Revert only when lift is at or below this value and the 90% interval lies entirely below zero. -0.1 means -10%. A drop by itself is not a revert. Absolute value must be at most 5",
"type": "number",
"minimum": -5,
"maximum": 5
},
"min_denominator": {
"description": "Minimum denominator unique users in each window",
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
},
"min_goal_events": {
"description": "Minimum unique goal converters across both windows when metric is omitted. When metric is set, numeric values (n) on metric_event across both windows. The field name stays min_goal_events",
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
}
}
},
"metric": {
"description": "Numeric property metric, for example p75(cwv_lcp) or sum(revenue). One of sum, avg, p50, p75, p95. Omit to keep the conversion-rate compare",
"type": "string",
"pattern": "^(sum|avg|p50|p75|p95)\\([A-Za-z0-9_]{1,128}\\)$"
},
"metric_event": {
"description": "Event whose property is measured. Defaults to goal_event. For hosted Web Vitals such as p75(cwv_lcp), set this to page_view",
"type": "string",
"minLength": 1,
"maxLength": 256
},
"better": {
"description": "Which direction is an improvement. Defaults to higher and is not inferred from the property name. Use better=lower for cwv_lcp and the other Web Vitals",
"type": "string",
"enum": [
"higher",
"lower"
]
}
},
"required": [
"project",
"goal_event"
],
"$schema": "http://json-schema.org/draft-07/schema#"
}