PRX Part 4: Our Data Strategy

Welcome back! This is Part 4 of the PRX series. Parts 1 to 3 covered model architectures, training design, and a 24-hour speedrun. This time we're pulling back the curtain on the part that quietly underpins all of it: the data. Of all the things that shaped PRX's quality, the data pipeline was one of the least glamorous parts to build but nevertheless an important piece to get right. Here's what we did, what we'd do differently, and a few things we only learned the slow way.

In one sentence: we assemble training data from a mix of public and internal datasets, re-caption the images with a VLM, and turn the result into the streamable corpus we trained PRX on.

At a high level, the data pipeline looks like this:

In the following we will dive into it in detail.

1. Guiding principles

A diverse dataset for pre-training

The goal was to assemble a large, diverse dataset for pre-training. At this stage the model is learning how the world looks: the visual concepts, the objects and scenes, how things are composed and lit, and the sheer range of what images can contain. That is a problem of coverage and diversity, not of per-image perfection. A broad, representative corpus teaches the model far more about the structure of the visual world than a smaller, prettier one would, even if many of the individual images are ordinary snapshots or slightly compressed. Over-filtering for aesthetics at this stage would actually hurt, narrowing the distribution and costing the model concepts and compositional variety it cannot recover later. Making generations look polished is a separate, later concern, which we leave to fine-tuning and preference alignment on small, ruthlessly curated sets. Pre-training is for breadth; fine-tuning is for taste.

A mix of data sources

Our captions philosophy

Data formats

These two formats play off each other throughout this post and the PRX data pipeline: Lance to build, MDS to stream.

On text latents

On image encoding

2. Building the dataset in Lance

System prompt:

Models