qwen-mm-plugins-api

Use omni_multi_speaker_asr, which returns speaker labels, timestamps, and an SRT file from an audio or video file. Pass num_speakers if known, or use transcribe_audio for fast chunked transcription of long files without diarization.

grounding performs spatial localization, returning 0-1000 normalized bounding boxes for objects in a single image. omni_av_grounding performs temporal localization, returning time segments in audio or video matching a natural-language query.

Yes. Every tool accepts base_url, api_key, and model arguments for any OpenAI-compatible endpoint. Configure OPENROUTER_API_KEY or ORCAROUTER_API_KEY and pass the matching base_url and gateway model ID; the server selects the key automatically.

Run references/launch_sam3_server.py with the sam3 package, its checkpoint, and BPE vocab available, plus fastapi, uvicorn, pillow, and numpy installed. It starts a multi-GPU FastAPI server; point SAM3_SERVER_URL at it and the segmentation tool works.

On DashScope endpoints, oversized local media uploads to model-bound temporary OSS up to 1 GiB, then falls back to user-managed OSS, then local frame sampling. Videos exceeding the model's server-side duration limit skip uploads and use local sampling directly.

Prefer perceive_media for general audio-video reasoning since it passes your prompt unchanged to the Omni model. Use the atomic tools like omni_asr or omni_av_counting only when their fixed structured output schema is explicitly required.