Assertions¶

A Tier 2 eval succeeds when its assertions pass. Each assertion is one graded condition checked against the agent's final response and the throwaway workspace it left behind. There are two families:

• Deterministic checks — file_exists, file_absent, regex, not_regex, command, tool_call. These inspect files, output, exit codes, or observed tool calls. They are cheap, fast, and reproducible.
• The LLM judge — llm. A pinned model reads the assertion and the agent's work and returns a verdict. Reserve it for holistic or subjective checks the deterministic ones can't express.

Every assertion is one entry in an eval's assertions array:

{
    "id": "parametrize",
    "prompt": "Write parametrized tests for the parse function …",
    "files": ["files/src/myapp/kv.py"],
    "assertions": [
        { "type": "file_exists", "path": "tests/test_kv.py" },
        {
            "type": "regex",
            "path": "tests/test_kv.py",
            "pattern": "pytest\\.mark\\.parametrize"
        }
    ]
}

An eval must declare at least one expectations entry or one assertions entry, or loading fails validation.

How assertions are graded¶

• Each assertion is pass / fail / skipped. Most return pass or fail. A few return skipped when the check can't run — a command whose requires binary is missing, or a tool_call against a harness that can't report tool calls. Skipped assertions count neither for nor against the case.
• Order is deterministic. expectations (see the llm section) expand to llm assertions graded first, in authored order, followed by the authored assertions in order.
• Patterns are Go RE2 regexes — no backreferences or lookaround. In a JSON or JSONC file every backslash doubles (\\., \\s); YAML single-quoted strings keep one.

The closed set of types is file_exists, file_absent, regex, not_regex, command, tool_call, llm. Each is covered below with a realistic example drawn from the bitwise-media-group skills.

file_exists¶

Passes when path (workspace-relative) exists after the run. Use it to assert the agent created the file it was asked to.

{ "type": "file_exists", "path": "tests/test_kv.py" }

file_absent¶

The mirror of file_exists: passes when path does not exist. Use it to assert the agent didn't create something it shouldn't have.

{ "type": "file_absent", "path": "internal/version/doc.go" }

Here the skill should recognise that a single-file package already carries its package comment, so adding a separate doc.go would be redundant — the assertion fails if the agent adds one anyway.

regex¶

Passes when pattern matches. With a path, the pattern is matched against that workspace file's contents; without a path, it's matched against the agent's final response text. Patterns are compiled in multiline mode, so ^ and $ anchor to line boundaries.

{
    "type": "regex",
    "path": "tests/test_kv.py",
    "pattern": "pytest\\.mark\\.parametrize"
}

A path-less variant checks the agent's own report — for example, that it surfaced the command it ran:

{ "type": "regex", "pattern": "terraform validate" }

A regex with a path whose file is missing fails (there is nothing to match against).

not_regex¶

The negation of regex: passes when pattern does not match. Use it to assert an anti-pattern is absent.

{
    "type": "not_regex",
    "path": "tests/test_kv.py",
    "pattern": "unittest|TestCase"
}

This guards a pytest suite against stdlib unittest leaking in.

command¶

Runs run through /bin/sh -c in the workspace and passes when the exit code matches expect_exit (default 0). This is the strongest deterministic check — it runs the real toolchain over the agent's output.

{
    "type": "command",
    "run": "terraform fmt -recursive -check",
    "requires": "terraform"
}

Use cwd to run inside a subdirectory of the workspace, and chain a setup step before the check:

{
    "type": "command",
    "run": "terraform init -backend=false -input=false >/dev/null && terraform validate",
    "cwd": "modules/widget",
    "requires": "terraform"
}

Set expect_exit when success means a non-zero code — for instance, asserting a forbidden token is gone (grep exits 1 when it finds no match):

{ "type": "command", "run": "grep -rq 'TODO' src/", "expect_exit": 1 }

requires keeps suites portable: an eval that needs terraform or python3 skips cleanly on a machine without it instead of reporting a false failure. The command shares the eval's timeout_seconds.

tool_call¶

Passes when the agent actually invoked a matching tool during the run — inspecting behaviour, not just the final artifact. tool is a regex matched against each observed tool name; the optional pattern is matched against that call's JSON-serialized arguments. An MCP tool surfaces as mcp__<server>__<tool>.

{ "type": "tool_call", "tool": "^Bash$", "pattern": "terraform validate" }

This confirms the agent ran terraform validate rather than only claiming to. To assert it reached for a specific MCP tool:

{ "type": "tool_call", "tool": "mcp__github__create_pull_request" }

Tri-state: if the harness under test can't report tool calls the assertion is skipped; if it reports calls but none match, the assertion fails. Unlike regex, tool and pattern are not compiled in multiline mode.

llm¶

Graded by an LLM judge instead of a fixed rule. The judge is one pinned model (anthropic/claude-sonnet-5 unless the judge_model config key or --judge-model overrides it) regardless of the model under test, so verdicts stay comparable across providers; it is driven by whichever installed harness supports that model. All of a case's llm assertions are graded in one judge session: the judge reads the numbered assertion texts, the agent's final response, and the eval's expected_output (as context, never a separate check), may inspect the workspace, and returns a pass/fail verdict with a short evidence quote per assertion. The judge runs with the same tool posture as the eval agent — evolve's OS sandbox is the confinement, not a tool allowlist — which is why deterministic assertions always grade before the judge session runs (see How evaluations run).

{
    "type": "llm",
    "text": "The new tests cover the error paths, not just the happy path."
}

Reserve llm for genuinely subjective or holistic judgements; prefer a deterministic check whenever one can express the same condition — a case with any llm assertions pays for one judge session, and deterministic checks are free and reproducible.

Bare-string shorthand¶

A plain string anywhere in assertions is shorthand for { "type": "llm", "text": <string> }:

"assertions": [
  "The commit message follows Conventional Commits.",
  { "type": "file_exists", "path": "CHANGELOG.md" }
]

expectations¶

expectations is a top-level array of statements, each expanded into an llm assertion and listed before the authored assertions, in authored order. They're graded in the case's single judge session alongside any inline llm assertions, and each still gets its own verdict and evidence. Reports tag these with source: "expectation" so they're distinguishable from inline assertions.

{
    "id": "focused-tests",
    "prompt": "Add tests for the new validator and explain your choices.",
    "expectations": [
        "The summary explains why each test case was chosen.",
        "No production code under src/ was modified."
    ],
    "assertions": [{ "type": "file_exists", "path": "tests/test_validator.py" }]
}

expected_output is related but different: it's the author's prose description of success. The judge sees it as context, but it is never graded as its own assertion — so migrating a suite that only has expected_output keeps its pass rate.

A combined example¶

A single eval typically mixes families: deterministic checks pin the concrete artifacts, and one judge assertion covers the part only prose can describe.

{
    "id": "parametrize",
    "prompt": "Write parametrized tests for parse in src/myapp/kv.py, including the error paths, in tests/test_kv.py.",
    "files": ["files/src/myapp/kv.py"],
    "expectations": ["The agent explains which error paths it parametrized and why."],
    "assertions": [
        { "type": "file_exists", "path": "tests/test_kv.py" },
        {
            "type": "regex",
            "path": "tests/test_kv.py",
            "pattern": "pytest\\.mark\\.parametrize"
        },
        {
            "type": "regex",
            "path": "tests/test_kv.py",
            "pattern": "pytest\\.raises"
        },
        {
            "type": "not_regex",
            "path": "tests/test_kv.py",
            "pattern": "unittest|TestCase"
        },
        {
            "type": "command",
            "run": "python3 -m py_compile tests/test_kv.py",
            "requires": "python3"
        }
    ]
}

For the case around these assertions — the prompt, input files/fixtures, and the per-case run controls — see Behavioral evals. Every field above is validated by the evals JSON Schema; point your editor at it via a "$schema" key for completion and inline errors.