235.5 sRecognize the event.
Second door opening detected.
18.8 sAnticipate the risk.
Warn before a collision.
355 sHelp at the right moment.
Remember the missing tomatoes.Recent multimodal large language models have advanced video understanding, but existing benchmarks mainly evaluate whether models can interpret observed visual content. In real-world scenarios, assistants must additionally anticipate what assistance is needed and decide when to provide it. To address this gap, we introduce P³IBench, a benchmark for proactive video understanding in real-life streaming scenarios. It evaluates three complementary capabilities: daily perception, risk prediction and intervention. We benchmark 12 representative open-source and proprietary video-language models, including streaming and adapted non-streaming systems. Our evaluation shows that current models struggle to make well-calibrated response decisions. Finally, we construct additional proactive training data and train P³I-Assistant-4B, achieving improvements on several tasks, though gains remain uneven across capabilities. Together, these findings show that effective proactive assistance requires not only visual understanding, but also precise response timing and appropriate restraint.
AbstractOne initial instruction. An unfolding video.
The assistant decides whether and when to respond.
Figure 1 · Illustrative target responses. Later frames provide context and do not justify earlier responses.
Everyday monitoring, emerging risks,
and goal-conditioned assistance.
Seconds · exact min / median / max on each row
Section 3.1 · Each task uses its own construction and verification procedure.
Evaluate what to say, when to say it,
and when to stay silent.
Models observe only past and current frames and decide when to respond.
Both metrics follow the same three scoring steps. They differ only in how responses are matched to targets.
Evaluate response oj against its matched target gi: correct content earns more when timing is close.
c: correctness of predicted content ŷj against reference yi—rules (0/1), or LLM-judged [0, 1] for intervention.
tj, τi: response and target times, in seconds.
Take the best response score from Step 1, then divide by the number of matched responses to penalize repetition.
Oi: all responses matched to target i, including zero-scoring replies.
No matched response: the target scores Si = 0.
Average all target scores from Step 2 and multiply by 100. The matching rule determines which PRS variant is reported.
Match each response to the most recent target at or before its emission time.
A response emitted before the first target is assigned to that target with a score of 0. This response still counts in the denominator when computing the target score in Step 2.
Match each response to the nearest target in time, allowing early responses to match future targets.
N is the number of ground-truth targets in the sample. SiU and SiB are the Step 2 target scores under the respective matching rules.
Section 3.3 · Evaluation protocol and metric definition.
12 evaluated models, plus P³I-Assistant-4B.
No model is reliable across all three capabilities.
Nine of the 12 baselines score below 30 Overall PRS-B. GPT-5.6-Luna performs best overall, at 46.59 PRS-B. It has an advantage in risk prediction and intervention but still struggles with timely event and object perception. Intervention is particularly difficult: the best baseline achieves only 29.85 PRS-B.
These results highlight a persistent gap between conventional video understanding and proactive assistance. Models must understand an evolving scene, follow complex instructions, and respond at the right moment. Current systems still struggle to meet these demands together.
Section 4.2 · Table 2 · Baselines only; P³I-Assistant-4B is reported separately.
Adapted offline models often produce unnecessary repeated replies. They tend to keep answering as in static question answering. On Accident Warning, Qwen2.5-VL-3B-Instruct emits at least three replies in 53 of 100 GT windows. Its score rises from 21.27 PRS-B to 71.39 when the repetition penalty is removed, using the same outputs. This shows how redundancy can substantially reduce the benefit of a correct, timely reply.
Native proactive models tend to reduce repetition but can remain silent when a reply is needed. JoyVL and MMDuet2 produce no reply in 79% and 100% of Accident Warning GT windows, respectively (Figure 3). This may reflect imbalanced supervision for silence and informative replies, suggesting a need for better training of response decisions.
Event perception is substantially less timely than object perception. Among GT windows with a response, GPT-5.6-Luna's median first-response latency is 24.7 s for Event Monitoring versus 1.5 s for Object Monitoring. Understanding events requires following continuous, complex contexts; this weakness also affects timely prediction and intervention.
Models also tend to respond before the annotated target time. Qwen3-VL-8B-Instruct scores 29.21 PRS-U versus 68.83 PRS-B on Accident Warning, where PRS-B allows early replies to match future targets. Early warnings can be appropriate, but the gap suggests difficulty locating the boundary at which a response is required. Both metrics are therefore important for assessing temporal alignment.
Section 4.2 · Table 2 & Figure 4 · Latency statistics exclude GT windows with no response.
Longer videos and later events are more challenging. The analyses of Qwen3-VL-4B-Instruct and JoyVL show this pattern, likely because the accumulated context becomes more complex (Figure 5).
A higher frame rate can help object perception while hurting risk prediction. For Qwen3-VL-4B-Instruct, increasing sampling from 1 to 2 FPS raises Object Monitoring PRS-B from 33.82 to 42.55, but lowers Accident Warning from 41.78 to 30.02.
Retaining more history can help. In the separate history-window ablation, Accident Warning PRS-B rises from 20.09 with one historical frame to 41.78 with all frames. Together, the ablations suggest that current models handle retained visual history more effectively than richer dynamic input arriving at each step.
Targeted post-training helps.
The gains remain uneven across tasks.
Accident Warning improves substantially, while Event Intervention remains difficult. Overall gains do not imply progress on every task.
Table 5 · PRS-B · Base → SFT → RL| Task | Base | + SFT | + RL |
|---|
Real response traces reveal failures
that a correct answer alone cannot explain.
Appendix D · Figure 13 · Instructions and reference targets are abridged. Later frames provide context and do not justify earlier responses.
Explore the paper, benchmark dataset, and source code.
