TimeScope: How Long Can Your Video Large Multimodal Model Go?

TL;DR

TimeScope is an open-source benchmark designed to measure how well vision-language models understand long videos. By adding short “needle” clips into videos ranging from 1 minute to 8 hours, it evaluates three skills:

• localized retrieval,
• information synthesis,
• fine-grained temporal perception. Timescope reveals that many state-of-the-art models still struggle with true temporal comprehension.

Table of Contents

• Why TimeScope?
• Benchmark Design
• Baseline Evaluation Results
• Open-Sourcing

Recent advances in multimodal AI have produced models claiming to understand hour-long videos. This trend mirrors progress in long-context language models, which excel at reasoning over lengthy text. Following this, vision-language systems now advertise context windows that can handle thousands of frames. But these claims require a closer look: do these models truly demonstrate understanding of the sequence of events? Are they limited to surface-level retrieval \ recognition? It's crucial to ask if their capabilities are being overstated.

Text benchmarks such as HELM and RULER have exposed the fragility of long-context claims, showing that models often struggle when tasks demand more than simple retrieval, like reasoning or aggregation at long context lengths. In the video domain, however, we're still playing catch-up. The most common test, Video Needle in a Haystack (VideoNIAH), injects static images as "needles" into videos, effectively measuring visual search rather than true temporal dynamics. As a result, even top-tier models advertising massive frame capacities are rarely trained beyond ~256 frames and see sharp drops on benchmarks like Video-MME when pushed further.

Why TimeScope? Motivating a Better Benchmark for Video

TimeScope flips the script by emphasizing three pillars of long-video understanding:

Benchmark Design

We evaluate across three needle types, each targeting a different aspect of long-video comprehension: