movie-annotation

To extract frame-level features for EEG encoding models from movies, you need low-level statistics, depth, motion, and semantic representations mapped to each frame. This pipeline processes MP4 files by running CPU extraction and GPU inference for visual models, outputting per-movie Parquet tables and NPZ files.

For naturalistic video analysis, you can compute low-level luminance, contrast, color, texture, monocular depth, optical-flow motion, scene-cut flags, face counts, and object categories. It also generates CLIP embeddings and zero-shot scene category scores for each frame.

Running CLIP and DPT inference for video annotation requires a GPU, as the pipeline uses batched GPU inference for these models. It also runs deterministic CPU feature extraction for low-level image statistics, combining both to process frame chunks.

The best way to batch process multiple MP4 videos for frame-synchronized features is using a reproducible chunked pipeline. This approach iterates through movie frame chunks, applies CPU and GPU inference, and saves per-movie outputs for downstream analysis.

Yes, you can use OpenCV and PyTorch together to extract depth and object detections from movie frames. The pipeline combines OpenCV DNN for face detection with Torchvision DPT for depth and DETR for objects, running them in a batched workflow.

Extracting EEG-relevant visual features from movies generates a per-movie Parquet table containing scalar feature columns and an accompanying NPZ file with CLIP frame embeddings. These outputs provide structured data for downstream EEG modeling.