Verify Harbor tasks in a separate sandbox

Starting today, you can verify Harbor tasks in a separate sandbox from the one used by the agent.

In task.toml, you can specify artifacts to copy between the agent sandbox and verifier sandbox.

Verifying in a separate sandbox:

• enables different resource configurations or dependencies than the agent sandbox
• lets users pre-bake dependencies into a verifier image to avoid flaky package installation
• provides an additional security boundary between the agent and the verification process

It also introduces constraints:

• verification is limited to copied artifacts, not full container state such as running processes
• tasks must explicitly write anything the verifier needs to an artifact path

Separate verification works for both single-step and multi-step tasks. Multi-step tasks can opt into a separate verifier sandbox per step.

To start using this feature, update the create-task skill:

npx skills add harbor-framework/harbor --skill create-task

Or add the following sections to your task.toml:

artifacts = ["/tmp/answer.json"]

[verifier]
environment_mode = "separate"

You can also configure the verifier sandbox:

artifacts = ["/tmp/answer.json"]

[verifier.environment]
cpus = 2
memory_mb = 4096
allow_internet = false

For multi-step tasks, configure the verifier per step:

[[steps]]
name = "grade"
artifacts = ["/tmp/answer.json"]

[steps.verifier]
environment_mode = "separate"

Or configure the verifier sandbox for a specific step:

[[steps]]
name = "grade"
artifacts = ["/tmp/answer.json"]

[steps.verifier.environment]
cpus = 2
memory_mb = 4096
allow_internet = false

Opting in treats the tests/ directory as an environment directory, similar to environment/.

As usual, we're excited to see what you build and welcome feedback!