AMD article finds world-model RL gains depend on repair and recipe

AMD’s technical article says reinforcement learning inside a robot world model becomes unreliable when the model mislabels failures as successes, and that both data repair and training-recipe fixes were needed to make gains reproduce across seeds. The report, published on 20 October 2026, focuses on the RoboTwin adjust_bottle task and argues that outcome fidelity matters more in a world model used as an RL environment than in one used only as an evaluator.

What the report tested

According to AMD, the study used BWM, an action-conditioned video world model, as the environment and π0.5 as the policy. The authors measured how often the model judged failed simulator rollouts as successful, then retrained the world model on failure rollouts and reran PPO inside the repaired model. They say the base BWM drew 82.5% of simulator failures as successes, a gap that pixel-level metrics barely reflected.

The article says failure-data fine-tuning cut that fake-success rate to 26.3% while preserving success recall at 0.964. The repair used 677 rollout trajectories from π0.5 and several simulator-PPO checkpoints, including 165 failures, and changed the data only, not the architecture or loss. AMD says the result suggests that broad visual similarity is not enough when the model is used to supply reward during policy optimization.

PPO recipe changes and reproducibility

AMD reports that a corrected PPO recipe inside the fine-tuned world model fixed three defects: truncation bootstrapping at the episode limit, the point where the critic attaches, and the value learning rate. With those changes, five training seeds all beat supervised fine-tuning when evaluated back in the simulator, with a mean gain of +9.7 percentage points and a standard deviation of 4.0. Under the original recipe, the mean gain was +5.3 points with a standard deviation of 14.3, and one seed fell to −19.0 points.

The report says the corrected runs matched SFT-level mismatch rates in closed-loop replay, while the degraded original-recipe seed appeared to exploit fake successes in the world model. AMD also states that the corrected recipe’s stability did not come from a more accurate critic alone: the critic’s explained variance remained near zero, but the recipe fixes prevented returns from drifting and removed the instability seen in the default setup.

What remained unresolved

Even after repair, AMD says some outcome errors persisted in action-fork cases, where the result depends on action details from a shared initial state. In those cases, the world model still drew 29% to 40% of failures as successes after retraining, and adding more on-policy data did not fix the issue. The article says these ambiguous states are exactly where reinforcement-learning gradients originate, which makes them a likely ceiling on future in-world-model RL gains.

The report also notes limitations. It covers one task, one policy and one world model, with only 10 policy updates per run. The author says the three PPO fixes were not individually ablated against simulator gains, and some initial states depend on sub-pixel differences that are not visible in the model inputs. The work is positioned alongside recent research on failure-aware world models, including CureWM and FACT, which address related failure-insensitivity problems with different training strategies.