Regrade

Treat regrade as a predictive feature: a regrade says, given this lock.json and this source trial, this is what we think the trial result would be in the new <trial-name>/ directory. The recorded agent execution is held fixed; only the verdict is recomputed.

Regrading re-scores completed trials with a new (or updated) verifier. The agent phase is never re-run: Harbor seeds a fresh trial with the recorded agent logs and artifacts from the source trial, then runs the new verifier against them in a separate verifier environment. This makes it cheap to fix a broken grader, tighten test cases, or compare verifier variants across the same agent executions.

Source trials and jobs are never modified. Every regrade produces a new trial or job directory with its own results.

Quick start

Regrade a job (can contain 1 trial as well):

harbor job regrade jobs/2026-07-16__15-44-21 -p ./my-task-v2 -e modal

Regrade a trial:

harbor trial regrade jobs/2026-07-16__15-44-21/my-task__abc1234 -p ./my-task-v2 -e modal

Both commands also accept a Harbor hub UUID instead of a local directory; the source is downloaded first (cached under trials/.sources/ or jobs/.sources/) and then regraded the same way:

harbor trial regrade 8f3c2a9e-1b4d-4c6f-9e2a-7d5b8c1f0a3e -p ./my-task-v2
harbor job regrade deeac1d1-9588-422f-8d8a-d504ec9fae14 -p ./my-task-v2

-p points at the task providing the verifier to regrade with, typically a copy of the original task with an edited tests/ directory. It accepts either a task directory or a directory of task directories (for example a downloaded dataset); tasks are matched to trials by task name.

At the end of a job regrade, Harbor prints a delta summary against the source rewards:

Regrade delta over 1 trial(s): 1 changed (0 up, 1 down), mean reward 1.000 → 0.500

How it works

For each source trial, Harbor:

1. Creates a new trial directory and copies the source trial's agent/ and verifier inputs (artifacts/) into it.
2. Builds the new task's verifier environment (from tests/) and uploads the recorded artifacts to their original container paths.
3. Runs the verifier and writes a fresh verifier/ directory and result.json.

No agent environment is started and the recorded agent is never re-instantiated, so regrading needs no agent credentials or API keys. A regrade is best thought of as a fork of the source trial: the new trial's config.json and lock.json carry the source trial's original agent configuration (and skill locks, copied verbatim from the source lock rather than re-resolved), so downstream consumers such as leaderboards keep seeing the trial's real inputs. The source trial's agent identity and token/cost stats are likewise preserved on the new result so results tables group correctly; the recorded cost is reported, not re-incurred (avoid summing cost across a job and its regrades).

Requirements

A trial is regradable when the new verifier's declared inputs are present in the record:

• The source is a completed single-step trial with a readable result.json and artifacts/manifest.json.
• The new task resolves to a separate-mode verifier ([verifier] environment_mode = "separate", or a [verifier.environment] table). Shared-mode verifiers inspect the live agent environment, which no longer exists.
• Every artifact the new task declares (including the implicit /logs/artifacts convention directory) has a manifest entry in the source trial, and its collected bytes still exist on disk at the path the replay reads.

Whether the source trial originally ran a shared or separate verifier does not matter; only the recorded artifacts do. Separate verifier mode is the recommended way to author tasks and will soon become the default. Trials that fail these checks are refused with a specific reason (recorded as per-trial errors in a job regrade; other trials proceed). Entries recorded as failed or skipped (a host-path collision) make the declaration incompatible, since the record does not contain trustworthy bytes for them.

Multi-step tasks and trials are not supported yet.

Options

Both commands share the core flags:

harbor job regrade also accepts -n/--n-concurrent and --job-name; harbor trial regrade accepts --trial-name. New trials get independently generated names; the link back to the source lives in provenance, not the name.

Provenance

Each regraded trial records where it came from:

• Trial level: config.json records a source_trial block, {action, type, trial_id, path}. action names the derivation (regrade today; other derivations may exist later) and type is local or hub: a hub source records only the trial UUID, a local source records the trial directory path plus its UUID when known. lock.json records the same block plus one extra field, task: the source trial's own task lock (name, type, and content digest) copied verbatim from the source lock.json; resolution also fills in the UUID for local sources that did not record it. The lock's verifier block records the resolved mode as verifier.environment_mode, and the result records it as verifier_environment_mode; source identity lives in the config and lock, not in result.json.
• Job level: the regrade job's config.json records source_jobs, a list of blocks with the same shape, {action, type, job_id, path} (the CLI passes one source; multiple source jobs are supported programmatically). Nothing extra is recorded in the job-level lock.json: the config pins the source identity and each derived trial's lock carries its own source_trial.
• A hub job regrade types every derived trial hub: both config.json and lock.json record only {action: "regrade", type: "hub", trial_id} per trial, and the job lock's trial entries match each trial's own lock.json exactly. The job archive is downloaded once and its trials are seeded into a scratch per-trial cache (<job_dir>/.sources/), which is removed when the regrade completes; an interrupted job keeps it so resume does not re-download, and a missing seed falls back to a per-trial hub download.
• Lock equality (used by harbor job resume to decide whether a rebuilt job is the same experiment) is anchored on the source trial's UUID and task digest, not the path: moving a source directory does not break resume, but swapping in a different trial at the same path is caught.

An interrupted job regrade resumes like any other job with harbor job resume.