SDK issue resolution
The SDK issue harness measures proposed patches with and without Loom code lookup. Both arms use the same model, system instructions and local search, read and exact-edit tools. The treatment adds one binding to the installed Loom Python SDK. Neither agent can execute tests or shell commands. A separate evaluator grades both patches with the same selected tests and timeout.
Both local search tools accept context_lines from 0 to 20 and return at most 10,000
characters. A literal lookup can therefore supply enough surrounding code for an edit
without requiring a second read.
The conditional instruction profile uses indexed definitions, relationships and concepts first when lookup is available. Literal searches and working-tree changes use local tools; an insufficient index permits local fallback. Supplied evidence is reused. No tool call is required merely to demonstrate adoption. Zero-use and failed runs remain in the result set. The treatment lists its indexed lookup first, followed by the same local tools in their original order; the manifest records this instruction and ordering profile.
This protocol exercises the code-index integration. It performs no memory warm-up or reuse, measured-coverage import or test-gate evaluation. It cannot establish whole-product efficacy or isolate a plugin problem. The target is correct patches with lower time and token use; neither availability nor this instruction profile guarantees adoption or a win.
Prepare inputs
Install the repository’s aws extra for boto3, and install sdk/python into the same
environment with uv pip install --python .venv/bin/python -e sdk/python. The harness
uses the ordinary loom.Agent import and existing cloud IAM credentials. It adds no SDK
provider adapter. Git and ripgrep must be available. Process supervision uses POSIX
process groups.
Use an isolated clean checkout at the exact issue base for each arm. Keep evaluator files, patches and output directories outside both agent checkouts. A sanitized task contains only public metadata and the issue text:
{
"instance_id": "owner__repository-issue",
"repo": "owner/repository",
"base_commit": "<full commit SHA>",
"problem_statement": "<public issue text>"
}
Optional public fields are version, environment_setup_commit and created_at.
Unknown fields, including reference and test patches, are rejected. Public task data is
available from SWE-bench Verified.
Selecting this dataset does not make a native evaluator an official-container score.
Prepare a private grader configuration shared by both arms:
{
"command": [
"python",
"<absolute evaluator script>",
"--patch",
"{patch}",
"--report-to",
"{output}"
],
"selected_tests": ["tests/test_behavior.py::test_case"],
"timeout_s": 120
}
The command runs outside the agent process, without a shell. Supported substitutions
are {patch}, {output}, {run}, {arm} and {instance_id}. The evaluator applies the
patch in its own clean checkout and keeps held-out data outside agent inputs. Its working
directory is the run’s grading/ directory; use absolute evaluator and dependency paths. Establish
base failures and reference-patch passes before inference. Freeze dataset selection,
dependencies, index tier and per-issue index commits in the experiment plan. The treatment
checks its live index commit before its first model call. Static edges remain lower bounds.
Run and grade
Model calls spend provider money. Obtain approval for the issue count, repeats, model
and arms before running. Index preparation and backend inference have separate costs.
Use the same flags for both arms except --arm, --checkout and --out:
PYTHONPATH=$PWD .venv/bin/python -m bench.sdk_issue_resolution run \
--task task.json --checkout <clean checkout> --expected-base <full commit SHA> \
--arm baseline --model <Bedrock model ID> --region <region> \
--org <org> --namespace <code namespace> --base-url "$LOOM_BASE_URL" \
--token-env LOOM_API_KEY --grader-config grader.json --out <unused run directory>
The Loom arm uses --arm loom. Token values are read from the named environment
variable and are absent from the manifest. A run directory is created exclusively;
existing runs cannot be overwritten or silently retried. The manifest freezes task,
instructions, tool schemas, dependency versions, runner hashes, limits and grader plan
before inference. The default agent timeout is 110 seconds; give an outer resource
supervisor enough additional time for artifact finalization. No provider or SDK transport
retries or prompt-cache checkpoints are configured.
Run the evaluator after each arm finishes, including empty or partial patches:
PYTHONPATH=$PWD .venv/bin/python -m bench.sdk_issue_resolution grade --run-dir <run directory>
PYTHONPATH=$PWD .venv/bin/python -m bench.sdk_issue_resolution compare \
--baseline <baseline run directory> --loom <Loom run directory>
The evaluator must write {output} as JSON with explicit valid, resolved,
selected_test_statuses and missing_tests fields. Statuses are keyed by the frozen
selectors and use PASSED, FAILED or ERROR. If a dataset selector abbreviates parameter
IDs, the evaluator must retain the mapping to actual collected nodes and require all
matching outcomes to agree. Missing results, missing selected tests, skipped outcomes,
timeout and invalid grading are invalid_setup, never a pass. Exit zero alone is not
resolution. A valid unresolved report stays unresolved.
Read artifacts
MANIFEST.json is the pre-inference protocol. events.jsonl preserves model responses
and all reported usage fields, failed tool results, SDK request IDs, lookup wall time and
returned evidence. agent.diff preserves tracked and new-file changes even after a
provider error, permission refusal, turn limit or supervised timeout. RESULT.json
separates execution status from patch presence and external resolution. A completed
empty patch is labelled completed_empty_patch; resolved remains null there.
stage, preflight_status and preflight_failed identify a binding preflight failure.
agent_inference_started: false records a failure before the agent inference loop;
true marks a loop attempt, and null means progress is unknown. Missing response usage
does not establish zero billing for an attempted provider call.
SDK HTTP 401, 403 and 409 refusals and local permission errors have guard_refused
status, including refusals from the index preflight. Cleanup errors retain an earlier
guard refusal. Tool-time SDK refusals retain their HTTP status, code and request ID.
Patch collection errors stay in patch_errors without replacing a refusal. A refusal
originating in cleanup records provider_cleanup or sdk_cleanup as its stage.
Recorded terminal, cleanup and denied-tool evidence keeps the first refusal primary
even when supervision later times out, is interrupted or reports a nonzero exit.
supervision retains that outer status, return code and error separately. A successful
terminal event cannot override failed supervision when no refusal was recorded.
A controller must stop before grading or starting another trial; it
must also check the serialized error.http_status when reading results from another
worker version. A failed preflight cannot establish a paired performance result.
The refusal does not authorize switching credentials, scopes or retrying the action.
Setup refusal leaves an empty patch and preserves the checkout. When a lookup has no
result, sdk_lookup_wall_s is null and sdk_lookup_timing_complete is false.
sdk_completed_lookup_wall_s retains the measured completed-call sum;
sdk_unfinished_lookups counts calls whose duration is unknown.
GRADE.json stores external grading separately. A hard process kill can leave a partial
last event; complete earlier events are retained.
Report every attempted run, including zero adoption, errors and invalid grading. Preserve cache-read and cache-write fields alongside ordinary input/output counts. Agent model usage is separate from SDK lookup latency, backend model/embedding usage and cold index preparation. Do not call agent-only usage total system cost. Compare only matching protocols; the comparison command rejects mismatched tasks, instructions, models, grader plans or limits. Repeated runs of one easy issue do not establish general benefit. Even a complete split from one project is a small sample, and repeated attempts on the same issue are not independent tasks. Keep negative, zero-use and failed attempts in their original cohorts. A setup or instruction change starts a separately frozen cohort.