CinePile 2.0 - making stronger datasets with adversarial refinement

In this blog post we share the journey of releasing CinePile 2.0, a significantly improved version of our long video QA dataset. The improvements in the new dataset rely on a new approach that we coined adversarial dataset refinement.

We're excited to share both CinePile 2.0 and our adversarial refinement method implementation, which we believe can strengthen many existing datasets and directly be part of future dataset creation pipelines.

If you are mainly interested in the adversarial refinement method, you can jump directly to the Adversarial Refinement section.

Wait. What is CinePile?

In May 2024, we launched CinePile, a long video QA dataset with about 300,000 training samples and 5,000 test samples.

The first release stood out from other datasets in two aspects:

• Question diversity: It covers temporal understanding, plot analysis, character dynamics, setting, and themes.
• Question difficulty: In our benchmark, humans outperformed the best commercial vision models by 25% and open-source ones by 65%.

Taking a look at a data sample

Part of the secret sauce behind it is that it relies on movie clips from YouTube and Q&A distilled from precise audio descriptions designed for visually impaired audiences. These descriptions offer rich context beyond basic visuals (e.g., "What color is the car?"), helping us create more complex questions.

Tell me more. How did you put together the original dataset?

To automate question creation, we first built question templates by inspecting existing datasets like MovieQA and TVQA. We clustered the questions in these datasets using a textual similarity model WhereIsAI/UAE-Large-V1 and then prompted GPT-4 with 10 random examples from each cluster to generate a question template and a prototypical question for each:

Using this approach, we generate approximately 32 questions per video. Prior to releasing CinePile, we implemented several mechanisms to ensure the quality of the dataset/benchmark that we cover in the next section.