Open TRL and OpenEnv Pipeline Trains Qwen3.5 to Paint Watercolours
An open Hugging Face pipeline uses TRL, OpenEnv and GRPO to train Qwen3.5 code models to paint watercolours, revealing gains, compute costs and taste risks.
Summary
On September 3, 2026, Sergio Paniego released an open reproduction of Surya Narreddi’s reinforcement learning watercolour project, whose August 23 video had topped 1.5 million views. The Hugging Face pipeline uses TRL, OpenEnv, Jobs, Spaces, Inference Providers and the Hub to train Qwen/Qwen3.5-35B-A3B with GRPO and LoRA. It generates roughly 150 lines of JavaScript through p5.brush, restricted to 10 of 47 methods, while a gate rejects sketches that fail, bypass the library, leave no pigment or manipulate scoring.
The 178-image, hand-rated hibiscus pool came from openly licensed iNaturalist photos and three vision-critic revisions: GLM-5.2 made 64, Kimi-K3 57, Qwen3-Coder-Next 35 and Qwen3.5-122B-A10B 22. Reward weights are 0.05 compilation gate, 0.05 length, and 0.90 split between open 7B preference model HPSv3 and Qwen3-VL-30B-A3B-Instruct comparisons against four references. Hps-only, weighted 0.90, rose from 0.58 to 0.71 over 60 steps; judge-led, 0.60 judge and 0.30 HPSv3, rose 0.45 to 0.72; hps-led, 0.30 and 0.60, rose 0.57 to 0.82, both over 110 steps.
Raising learning rate from 2e-5 to 5e-5, adopting constant_with_warmup scheduling, disabling group reward scaling and targeting all linear LoRA layers unlocked learning. Pairwise judging cut sub-0.3 rollouts from 99 to 16 and 37 to 4, and doubled paint coverage from 0.11 to 0.23 and 0.13 to 0.30. However, the model-only, single-subject pool drove outputs toward one flower. Each eight-rollout step took 15 to 18 minutes; an H200 needed 18 hours for 60 steps or 34 for 110. Infrastructure failures affected 1.5% of rollouts, peaking at 5.2%, while an OpenEnv websocket bug ruined two runs before an upstream fix. All artifacts are open.
Positives
- Hps-led achieved the strongest final mean reward, rising from 0.57 to 0.82 across 110 steps.
- Judge-led reduced rollouts scoring below 0.3 from 99 to 16 while increasing mean reward by 0.27.
- Pairwise judging doubled paint coverage from 0.11 to 0.23 and from 0.13 to 0.30 across the two judge runs.
- Every reference, sketch, rollout, adapter, training curve, environment, scorer and reproduction recipe is openly published.
- A submitted OpenEnv websocket fix eliminated the persistent connection failure that had ruined two runs.
- A 4B model produced valid sketches in side experiments, suggesting training costs could fall by roughly an order of magnitude.
Risks & concerns
- The 178-painting pool contains no human-made work and represents only hibiscus images, limiting both its aesthetic grounding and output diversity.
- Infrastructure failures corrupted about 1.5% of all rollouts and 5.2% in the worst run before failed requests were excluded.
- Each step required 15 to 18 minutes, with rendering consuming 70% to 80% of runtime and individual renders taking 69 to 96 seconds.
- Four paid services must remain healthy simultaneously, and the continuously running HPSv3 scorer can cost more than training.
- Hps-only converged on similar colours and an estimated 0.771 mean ceiling, although whether additional training could exceed it remains unknown.
- The reward depends on subjective model proxies and randomly sampled references, making scores noisy and tightly coupled to one curator’s preferences.