doubao-multimodal

What problem does it solve?

When you have long or complex audio/video (meetings, lectures, podcasts, interviews, demo videos), producing accurate transcripts, speaker logs, aligned subtitles, translations, and time-based structure usually requires multiple manual tools and workflows.

Core Features & Use Cases

• End-to-end multimodal understanding for ASR/AST/diarization: supports plain ASR, per-character timestamp ASR, multispeaker ASR, and diarization with time ranges.
• Subtitle alignment: aligns provided reference subtitles to produce character-level start/end timestamps for high-precision captioning and karaoke-style effects.
• Multimodal video-focused outputs: generates audio/visual markdown reports (caption), extracts video timeline JSON, and selects technical-blog keyframes (optionally guided by a transcript).
• Scales to oversized media: automatically downloads remote URLs or uploads local files to Volcano TOS via pre-signed URLs, splits long media, runs concurrent calls, and merges results with per-part timing.

Quick Start

Ask your agent to run the skill for a remote meeting clip and produce speaker diarization by using the command: bun run skill/scripts/main.ts --task diarize --url "https://example.com/meeting.mp4" --type video --audio-part-type input_audio --out-dir ./out/diarize.