Reviewing reports¶

Reports are the repository-level view over a sweep. evolve report reads every committed results.<ext>, rolls the entries up across plugins, skills, and models, and renders two artifacts: a human-readable EVALUATION.md and a machine-readable rollup. Nothing is measured here — the report is a projection of what the stored results already hold, so it re-renders identically until the results change.

Generating reports¶

evolve report is the last step of evolve run all, but it touches no agents, so you can re-run it any time the stored results change — after a partial sweep, a --migrate, or a models restriction.

evolve report                 # regenerate EVALUATION.md + the machine rollup
evolve report --migrate       # upgrade stored results to the latest schema first
evolve report --check         # also gate on configured thresholds (non-zero exit on breach)

The output format follows --results-format; switching formats removes the stale rollup from the previous choice. What gets written depends on the repository layout:

The Markdown report¶

EVALUATION.md opens with a generated-by marker, an # Skill evaluations heading, and a methodology paragraph explaining where the figures come from and how to read empty cells. If a models restriction is active, an ## Excluded models table follows, listing the catalog models left out (so a partial matrix never reads as the whole picture).

Then one ## <plugin> section per plugin, each with up to two rollup tables.

Rollup tables¶

The per-plugin Triggers rollup, one row per model:

The per-plugin Evals rollup adds the measured-usage columns:

A rendered slice of a single-layout report — the solo plugin section, with its Triggers and Evals rollups:

Triggers¶

Evals¶

Detail tables¶

The detail tables turn the view case-major — one heading per trigger query or per eval, with the models as rows, so comparing models on a single case is the default. In a single-layout repo they follow the rollups in the root EVALUATION.md; in multi/marketplace repos they move to each plugin's own EVALUATION.md.

Trigger detail, under a #### <query> (expected: yes|no) heading:

Eval detail, under a #### <id> — <name> heading, mirrors the eval rollup's columns but swaps the passed-count for a per-case Result: PASS, FAIL, ERROR (a runtime error — the agent run failed), or —. Below the table, each failed run is itemised — runtime errors as - <model> runtime error: <message>, and each failed expectation as - <model> failed <assertion>: <evidence> — so a failure points straight at the judge's reasoning.

The machine-readable rollup¶

EVALUATION.<format> is the same data as a structured object, for CI gates, dashboards, and diffing. Its top level:

Under each plugin, triggers and evals are maps of provider/model-id → a model rollup, and skills nests the same shape per skill for drill-down. Each model rollup carries the aggregates the Markdown tables render — passed, failed, errored, total, pass_rate, avg_run_seconds, the estimate and measured usage objects — plus the machine-only baseline summary and the previous_delta / baseline_delta objects (rate, avg_run_seconds, input_tokens, output_tokens, cost_usd) that the Markdown renders as Δ rate and Lift vs base.

Gating on thresholds¶

evolve report --check turns the rollup into a CI gate: it compares the trigger and eval pass rates against the thresholds in your .evolve config (or the --min-*-pass-rate overrides) and exits non-zero on a breach, printing each as FAIL: ….

evolve report --check --min-triggers-pass-rate 0.95 --min-evals-pass-rate 0.90

With no threshold configured and no flag passed, --check is an error rather than a silent pass. Exit codes follow the reference: 0 clean, 1 on a breach, 2 on a usage or runtime error.