Using Open-Weight LLMs to Edit Neural Network Policy Weights
Can an open-weight LLM adapt a trained control policy to changed dynamics by directly editing its weights? This experiment tests that question on a LunarLander PPO actor using GPT-OSS-120B—no Astra required.
The Setup
The target policy is small: eight observations, two 64-unit tanh layers, four action logits, and 4,996 parameters. GPT-OSS-120B inspects tensor rows and proposes edits—input rescaling, logit adjustments, hidden-unit scaling, or direct assignments. Each proposal starts from the original weights. The editor stays frozen; only the edited weights deploy.
Each run allows 12 candidates evaluated on 64 development episodes, using mean episodic return or landing success as the search objective.
Exact Repair: The Easy Case
With left and right thrusters reversed, GPT-OSS immediately proposes swapping the corresponding output rows and biases. The result: 130 changed parameters, success rate restored from 0/256 to 226/256. This works because the perturbation is a disclosed action permutation—the edit simply inverts it.
Adaptation: The Hard Case
Three harder tests reduced main-engine thrust, added crosswind, or combined thrust, gravity, and wind changes. Across 18 editing runs, four policies passed independent approval. None succeeded under combined perturbations.
The strongest result came from crosswind correction. The editor amplified horizontal position and velocity inputs and adjusted engine logits—387 weight entries in total. The paired outcomes show why even a passing result needs scrutiny: 32 landings gained, 17 lost, 15 net gains, with 199 episodes failing under both the original and edited policies.
Only this first crosswind edit produced a confidence interval strictly above zero. Passing approval did not guarantee a clear gain on the final test.
Give the Editor Better Tools
The core limitation is diagnostic: tensor inspection and rollout scores force the editor to guess which computation needs changing. Better inputs would narrow the search before episodes are spent testing bad edits.
The tools already exist:
- NNsight for activation interventions
- TransformerLens for transformer inspection
- Anthropic’s circuit-tracing and sparse feature tools for studying internal representations
- Activation patching to localize failures directly
For a small actor like this one, direct ablations and activation patching are the right starting point. The transformer-focused tools point toward a direction for larger policies.
Why This Matters for Hardware
Embodied policies make weight editing worth pursuing seriously. A weaker actuator or miscalibrated sensor can break a controller without invalidating everything it learned. If the failure localizes to specific computations, a targeted weight edit could preserve useful behavior and leave a compact policy running on the robot—with no LLM in the control loop at inference time.
The next experiment is straightforward: compare editors with and without interpretability tools under matched inference and simulator budgets, then verify the approach on physical hardware.