Direct Weight Editing of Neural Network Policies Without Astra

Astra demonstrated that a language model can read and edit neural network weights directly. This experiment replicates that idea using GPT-OSS-120B, an open-weight model, applied to a trained LunarLander policy under changed physics—no gradient updates required.

The Setup

The target policy is small: 4,996 parameters across two 64-unit tanh layers, with eight inputs and four action outputs. GPT-OSS inspected the weights, proposed edits, and received rollout scores as feedback. Each run allowed 12 candidates evaluated over 64 development episodes. The LLM called no inference during flight—once edited, the policy ran on its own.

Two objectives guided the search: maximizing reward, and maximizing landing success while preserving performance under original physics. A successful landing requires both legs on the helipad, the body at rest, and no crash within 1,000 steps.

Reversed Thrusters: A Clean Repair

The clearest result came from reversing the left and right thrusters. GPT-OSS’s first edit swapped the matching output rows and biases—130 parameters total—taking landing success from 0/256 to 226/256. The model received information about the reversal and exploited the exact inverse mapping. The edited policy requires the swap; it fails under original physics.

Harder Physics: Mixed Results

Three additional changes tested the limits: weaker main-engine thrust, crosswind, and a combined thrust-gravity-wind shift. Across 18 LLM editing runs, four policies passed independent approval. None passed for the combined changes.

The strongest result came from the crosswind condition. The approved edit scaled horizontal position and velocity inputs and adjusted engine logits, touching 387 weight entries through a small set of transformations. Landing success improved from 25/256 to 40/256—a gain of 5.9 percentage points (95% bootstrap interval: +0.8 to +11.3). The other three approved edits had final intervals that included zero despite passing approval thresholds.

That crosswind edit traded some existing successes for new ones: 32 landings gained, 17 lost, 15 net improvement. GPT-OSS didn’t select 387 individual values—it applied structured transformations. This does not indicate neuron-level understanding.

What the Results Don’t Show

Neither objective—reward or landing success—clearly outperformed the other; each produced two approved policies. The comparison used one target policy and three search seeds per condition, with provider constraints limiting total searches. A meaningful claim about objective choice would require comparison against PPO adaptation under the same simulator budget, repeated across independently trained policies.

What to Try Next

Rollout scores tell the editor whether an edit helped, not where to look. Ablations and activation patching could localize faults before editing begins, potentially enabling smaller models to make useful repairs. The practical appeal is a controller that adapts to sensor or actuator changes and then runs independently—no LLM in the loop during deployment. Whether this transfers to actual hardware remains untested.

The core finding stands: an open-weight model can repair a small RL policy through direct weight editing, replicating the central idea of weight-space programming without proprietary infrastructure.