Evaluation API

Evaluations select shadow decisions over a defined observation window and compare integration metrics with recorded success criteria. Use them to measure policy behavior, evidence coverage, and application outcomes. Paid evaluation terms are handled separately through your commercial agreement.

Define and close a study

Create the evaluation with the intended action keys and success criteria before interpreting results. Supported criteria concern shadow-decision counts, abstention, and outcome coverage. Supply timezone-aware boundaries where used. Keep policy and application changes documented alongside the evaluation.

The report includes the selected decision distribution, coverage, insufficient-evidence reasons, challenge and outcome metrics, operator-change indicator counts, latency summaries, and criterion results. The operator-change count is returned as experimental_operator_change_indicators and counts decisions carrying operator_change.detected, rather than independently verified attacks.

The result is passed whenever all recorded criteria are met. Otherwise it is in_progress for an open study or failed for a closed study. An open study can pass before closure; close it to fix the observation cutoff.

abstention_rate is the fraction of selected shadow decisions carrying decision.insufficient_evidence; it is not simply the fraction labeled hold or deny. false_challenge_proxy is the fraction of challenge decisions subsequently labeled executed or manually_approved. That proxy does not establish that those challenges were unnecessary. Outcome coverage is the fraction of selected decisions with a timely recorded outcome.

Closing fixes the cutoff to the earlier of the planned end and closing time; repeated closing preserves it. Later decisions and outcomes are excluded. Save the report before relevant trace retention removes underlying data required for your review.

Interpretation boundary

Interpret results with the selected sample, outcome coverage, and individual criteria. Your review process establishes the basis for customer-reported labels. Identify synthetic fixtures separately from participant observations and retain exclusions that affect how you apply the result to your workflow.

See Evaluate for protocol design and Commercial model for paid evaluation terms. The operation contracts below describe creation, listing, reporting, and closure.

GET /api/v1/evaluations

[
  {
    "id": "evl_example",
    "name": "Payout shadow study",
    "action_keys": [
      "payout.destination.change"
    ],
    "success_criteria": {
      "min_shadow_decisions": 50,
      "max_abstention_rate": 0.5,
      "min_outcome_coverage": 0.5
    },
    "status": "open",
    "starts_at": "2026-10-01T12:00:00Z",
    "ends_at": "2026-10-31T12:00:00Z",
    "created_at": "2026-10-01T12:00:00Z"
  }
]

POST /api/v1/evaluations

{
  "name": "Payout shadow study",
  "action_keys": [
    "payout.destination.change"
  ],
  "success_criteria": {
    "min_shadow_decisions": 50,
    "max_abstention_rate": 0.5,
    "min_outcome_coverage": 0.5
  }
}

{
  "id": "evl_example",
  "name": "Payout shadow study",
  "action_keys": [
    "payout.destination.change"
  ],
  "success_criteria": {
    "min_shadow_decisions": 50,
    "max_abstention_rate": 0.5,
    "min_outcome_coverage": 0.5
  },
  "status": "open",
  "starts_at": "2026-10-01T12:00:00Z",
  "ends_at": "2026-10-31T12:00:00Z",
  "created_at": "2026-10-01T12:00:00Z"
}

POST /api/v1/evaluations/evl_example/close

{
  "id": "evl_example",
  "status": "closed",
  "result": "failed",
  "ends_at": "2026-10-01T12:00:00Z"
}

GET /api/v1/evaluations/evl_example/report

{
  "id": "evl_example",
  "status": "open",
  "result": "in_progress",
  "metrics": {
    "decisions": 0,
    "abstention_rate": null,
    "outcome_coverage": 0.0
  }
}