Judge Criteria

Judge criteria let you use an LLM or agent CLI to evaluate work, configured via TOML files. This makes it trivial to reuse and share rubrics between tasks.

LLM judge

[judge]
judge = "anthropic/claude-sonnet-4-6"
files = ["/app/main.py", "/app/utils.py"]

[[criterion]]
description = "Is the code correct?"
type = "binary"

[[criterion]]
description = "How readable is the code?"
type = "likert"
points = 5
weight = 2.0

[[criterion]]
description = "Rate the test coverage on a scale from 0 to 100"
type = "numeric"
min = 0
max = 100

The judge field accepts any LiteLLM model string. It can be overridden at invocation time without editing the rubric. See Provider routing.

Agent judge

Agent judges shell out to a CLI like Claude Code or Codex. Unlike LLM judges, they can explore the filesystem and run commands:

[judge]
judge = "claude-code"
model = "anthropic/claude-sonnet-4-6"
isolated = true

[[criterion]]
description = "Does the solution handle edge cases?"
type = "binary"

MCP servers

Each [[judge.mcp_servers]] entry matches a Harbor task's [[environment.mcp_servers]]. Per-server allowed_tools lists the tools the judge may call; omit it to allow all of the server's tools. codex ignores allowed_tools and does not support sse servers.

[judge]
judge = "claude-code"

[[judge.mcp_servers]]
name = "playwright"
transport = "stdio"
command = "npx"
args = ["@playwright/mcp@latest", "--headless", "--isolated"]
allowed_tools = ["navigate", "click"]

[[criterion]]
description = "Does the rendered page match the spec?"
type = "binary"

Individual mode

Set mode = "individual" to grade one criterion per call instead of batching them all into one. LLM judges make one request per criterion; agent judges run one CLI invocation per criterion, sequentially. For LLM judges, each criterion can also scope its own files (below):

[judge]
judge = "anthropic/claude-sonnet-4-6"
mode = "individual"

[[criterion]]
description = "Is the analysis correct?"
files = ["/app/analysis.pdf"]

[[criterion]]
description = "Is the spreadsheet well-structured?"
files = ["/app/data.xlsx"]

Criteria without files fall back to [judge].files.

If a judge call times out, RewardKit records the affected criteria as 0.0 with an error and warning in reward-details.json.

Configuration reference

[judge] section

[[criterion]] entries

[scoring] section

Controls how the judge's criteria are aggregated into a single score. This only affects criteria within this TOML file — it does not change how programmatic and judge scores are combined across the directory.

[scoring]
aggregation = "all_pass"  # weighted_mean | all_pass | any_pass | threshold | required_pass
threshold = 0.7           # only used with "threshold" aggregation

required_pass returns 1.0 only when every non-optional criterion passes (value > 0); optional criteria never gate. With no non-optional criteria it warns and scores 0.0.

Score normalization

• Binary: yes/true/1 → 1.0, anything else → 0.0
• Likert: normalized to [0, 1] as (raw - 1) / (points - 1)
• Numeric: normalized to [0, 1] as (raw - min) / (max - min)

Negated criteria

Set negate = true for a criterion describing behavior the answer should not exhibit. The judge scores presence as usual, then the score is inverted (value → 1 - value): present → 0.0, absent → 1.0. The raw judge answer is kept in reward-details.json so the flip is auditable.

[[criterion]]
description = "States there is no task execution history tracking in the database"
type = "binary"
negate = true   # the answer should NOT make this (false) claim

Trajectory evaluation

To evaluate the agent's process rather than just its output, point the judge at the trajectory file:

[judge]
judge = "anthropic/claude-sonnet-4-6"
atif-trajectory = "/logs/trajectory.json"
files = ["/app/main.py"]

[[criterion]]
description = "Did the agent take an efficient approach?"
type = "likert"
points = 5

The trajectory content is truncated proportionally to fit within the model's context window while preserving all steps.

Custom prompt templates

You can provide your own prompt instead of the built-in one:

[judge]
judge = "anthropic/claude-sonnet-4-6"
prompt_template = "my_prompt.md"

The template must contain a {criteria} placeholder where criterion descriptions get injected.