OmniScript

Abstract

TL;DR — An 8B omni-modal model that converts long cinematic videos into structured, temporally-grounded scripts.

The Video-to-Script Task

Generating hierarchical, temporally-grounded scripts from long-form cinematic videos.

Figure 1: Overview of our Video-to-Script (V2S) framework. Given a long-form cinematic video, the pipeline performs temporally grounded scene-event parsing and generates a structured script with multimodal fields (dialogue, action, expression, and audio cues).

Given an untrimmed cinematic video containing complex narrative transitions, recurring characters, and multimodal cues, the model produces a structured output organized into three hierarchical levels:

Architecture

OmniScript: an omni-modal (audio + visual + language) model built on Qwen3-VL with AV-DeepStack injection.

Figure 3: Overview of the proposed architecture. Instruction, video, and audio are encoded into multimodal tokens and fused in the LLM via AV-DeepStack across multiple layers. The model first performs multimodal plot and character-relationship reasoning and then generates structured script outputs.

Progressive Training Pipeline

A four-stage training recipe followed by reinforcement learning refinement.

Figure 2: Overview of the memory-augmented progressive annotation pipeline. The character profile manager injects historical profiles to guide plot reasoning and dynamically updates character memory. The generated plot description and raw audio-visuals are then fed into a Gemini-based annotator to produce a fine-grained video script.

V2S Benchmark

A meticulously curated benchmark with hierarchical evaluation metrics.

Experimental Results

OmniScript (8B) outperforms much larger open-source models and rivals proprietary systems.

Event-Level Comparison on 5-Minute Videos

Scene-Level Comparison on 5-Minute Videos

Ablation Study: Training Strategy

Ablation Study: Subtitle Removal

Vision-only models rely heavily on burned-in subtitles for dialogue recognition. Removing subtitles reveals whether a model truly understands speech.

OmniScript for Long Video

Extending OmniScript from short clips to full-length cinematic videos (10–45 minutes).

To scale OmniScript from short clips to long videos, we investigate two complementary strategies that trade off between end-to-end global reasoning and modular controllability.

Comparison of two strategies for extending OmniScript to long videos. Left (Strategy 1): Direct long-context extension via pseudo long-video composition. Right (Strategy 2): Two-stage pipeline with plot segmentation followed by clip-level script generation and merging.

Multi-dimensional Performance Across Video Durations

Performance comparison across video durations (10, 20, 30, 40 min) on multiple metric dimensions including event-level (action, dialogue, character, expression, audio cue, tIoU) and scene-level (location, environment, mood, time, type, tIoU) metrics.

F1 Scores Across Long-Video Durations (10–45 min)

(a) Event-level, scene-level, and overall F1 scores across durations from 10 to 45 minutes. (b) Event-level fine-grained F1 scores on character, action, and dialogue. OmniScript-TSG (red) exhibits the strongest long-horizon robustness.

Visualization Examples

Explore OmniScript's structured script generation results on real cinematic videos.

Citation

If you find our work useful, please consider citing our paper.

@inproceedings{omniscript2026,
  title     = {OmniScript: Towards Audio-Visual Script Generation
               for Long-Form Cinematic Video},
  author    = {Pu, Junfu and Chen, Yuxin and Wang, Teng and Shan, Ying},
  journal   = {arXiv preprint},
  year      = {2026}
}