Training a coding model to paint watercolours with TRL and OpenEnv

On 23 August, Surya Narreddi posted a beautiful video of watercolours painted by a language model. The model writes JavaScript through p5.brush, a library that "adds natural drawing tools to p5.js". The video went viral fast, over 1.5M views at the time of writing.

The video came with a blog post explaining the training behind an earlier and narrower stage of the project, close-up flowers rather than the full compositions in the video, sadly without open artifacts yet. His site says a full technical report is coming, so ensure you follow him. The original idea is his, coming from the art and design side, where his skills are way beyond mine. My attempt is on the engineering side, reproducing the recipe in the open with every piece published.

Note: for the context behind the project, told by Surya himself, watch this video of his thesis.

In this article I try and reproduce his idea with TRL and OpenEnv. The reference pool dataset, the RL environment, the training scripts and the trained models, all open.

The whole pipeline runs on Hugging Face, end to end:

• training on Jobs
• the RL environment and the scorer model as Spaces
• the pairwise judge through Inference Providers
• and every artifact on the Hub, gathered in one collection

Once the two Spaces are up, the recipe is one command. Duplicate the environment and the scorer model, set two environment variables for the reward mix, and launch:

The rest of this article is the story of getting there, and every piece is in the repo.

Why people loved it

In that same period, Anna Ridler photographed thousands of tulips, hand-labelled every one, exhibited the dataset itself as the artwork, and later trained a model on it. I found her work through the references AI agents brought back while building this project and loved it because this project does something very similar by curating a set of images by hand, and then training against them.

RL over taste

The reward, as his blog defines it and as the RL environment I built implements it: