FineVideo: behind the scenes

Open video datasets are scarce and therefore slowing down the development of open-source video AI. For this reason we built FineVideo, a dataset with 43k videos that span 3.4k hours and are annotated with rich descriptions, narrative details, scene splits, and QA pairs.

FineVideo contains a highly diverse collection of videos and metadata which makes it a good ingredient to train models to understand video content, train diffusion models to generate videos from a text description or train computer vision models using its structured data as input.

Wait, you haven’t seen FineVideo yet? take a look at it through the dataset explorer page.

Table of Contents

• Table of Contents
• About this blog post
• Building the Raw dataset
  • Filtering YouTube-Commons
  • Downloading the videos
• Keeping dynamic content
  • Word density filtering
  • Visual dynamism filtering
• Video Categorization
  • Custom built Taxonomy
  • Content annotation
  • Feedback loop taxonomy - content annotation
• Contributing descriptive metadata
  • Long videos & Gemini 1.5 Pro
  • Content selection
  • Annotating with Gemini 1.5 Pro and Structured Output with GPT4o
• Fine Alignment and anomaly filtering
• Future Work

About this blog post

In this blog post, we share the technical details and code involved in developing FineVideo: a journey that starts with 1.9M videos in YouTube-Commons and ends with 44K videos with all details annotated.

A good way to start is taking a look at the different steps of our journey. Those steps involve content filtering, annotation and output structuring.

In the following sections we discuss each of the steps and provide references to relevant parts of the code. If you prefer to navigate the code directly, take a look at our FineVideo repository on Github.

First, let’s have a look how we got an initial list of YouTube videos and how we apply some first filters.

Building the Raw dataset

Filtering YouTube-Commons

Below some details on the filters and metadata fields that we keep:

Filters

Metadata fields

Click to Expand Metadata Fields

Downloading the videos