Harness Learning: Training LLMs to Adapt Agent Programs at Test Time
Language model agents depend on two components: the model itself and the harness—the executable program that organizes model calls, tool use, and information flow. Different tasks demand different harness structures, yet finding effective revisions through manual iteration is slow and brittle. Harness learning solves this by training a model to revise harnesses automatically, using execution feedback as its guide.
What Is Harness Learning?
Harness learning trains a proposer model to edit a solver’s harness code in response to execution outcomes. The framework treats this as meta-learning over executable programs:
- Outer loop: Train the proposer from revision outcomes across many tasks
- Inner loop: At test time, use the proposer to adapt the harness iteratively from execution feedback
At test time, both the proposer and solver remain frozen. Only the harness changes.
The proposer receives three inputs: the task description, the current harness, and an execution report. It outputs a SEARCH/REPLACE code edit. The revised harness runs on development questions, producing a reward (task performance) and fresh feedback for the next round.
Training the Proposer
Training proceeds in two stages:
Supervised fine-tuning (optional): A teacher model generates harness revisions on training tasks. Successful revisions—those that improve development scores and pass quality filters—form the SFT corpus. This initializes the proposer.
Reinforcement learning: The proposer samples candidate revisions, each scored on held-out questions disjoint from the feedback set. GRPO normalizes rewards within each group of candidates and updates the proposer toward higher-scoring revisions.
The reward combines task performance with validity bonuses for edits that parse and execute successfully.
Multistep RL extends this by training on successive revision states: after selecting a candidate, the system generates fresh feedback and treats the updated harness as the next training context.
Experimental Results
Two settings test whether learned revisions generalize.
Reasoning Gym
Training covers 26 task families (reasoning puzzles like maze, sudoku, tower of hanoi). The proposer is evaluated on 21 unseen families excluded from both SFT and RL.
Key results:
- Base proposer mean score on unseen families: 0.32
- After SFT + RL: 0.62
- The trained 4B proposer outperforms its 35B teacher on average (0.62 vs. 0.56)
- Five sequential revisions match or exceed the oracle best-of-eight independent proposals
Multi-Hop Question Answering
A proposer trained on HotpotQA transfers to MuSiQue and 2WikiMultihopQA—benchmarks unseen during training.
Key results:
- Single-step RL on MuSiQue: 0.15 exact match (near seed)
- After ten sequential revision rounds: 0.27
- Average final run of each RL proposer exceeds oracle best-of-80 independent proposals
Sequential revision substantially outperforms independent sampling, especially on QA tasks where the seed harness already contains a multi-hop retrieval workflow with multiple refinable components.
What the Proposer Learns
Training improves proposal reliability: RL reduces failed or zero-scoring proposals from ~38% (Base) to ~8% on unseen Reasoning Gym families. More viable candidates per round sustain progress across successive revisions.
Qualitative inspection reveals consistent structural changes:
- Reasoning Gym: Proposers frequently introduce interpreter loops, delegating computation to the Python runtime
- QA: Proposers keep summaries for guiding retrieval while passing raw passages directly to the answering call, separating query formulation from answer generation. Other revisions add a third retrieval hop or trigger re-search when a draft answer is absent from retrieved passages
These structural changes transfer to unseen tasks, suggesting the proposer learns reusable adaptation skills rather than task-specific patches.
Does Multistep Training Help?
Training on revision sequences does not consistently outperform training on individual revisions. On Reasoning Gym, Multistep RL beats Single-step RL on canonical questions but falls below it on format-varied questions. On QA, both configurations reach nearly identical final scores.
Notably, proposers trained only on individual revisions continue improving harnesses across multiple test-time rounds. Iterative improvement emerges from single-revision training—multistep training is not required to enable it.
Limitations and Next Steps
Current limitations include:
- Supervision limited to one teacher model per setting
- The proposer cannot choose its own inspection or testing steps before submitting a revision
- Proposer and solver train independently, leaving joint optimization unexplored
Future directions include training tool-using proposers that inspect code and run targeted tests before revising, using stronger or more diverse teachers for initialization, and jointly optimizing proposer and solver while preserving transfer to unseen tasks.
Harness learning demonstrates that a model can learn how to improve executable agent programs—not just what to output—and that this adaptation capability transfers to tasks the model has never seen.