tao-generate-image-grounding

Extract referring expressions and bounding boxes from image-caption pairs using Vision-Language Models.

83|20|Updated Apr 16, 2026
One-click install
npx skills add https://github.com/NVIDIA-TAO/tao-skill-bank --skill tao-generate-image-grounding
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: tao-generate-image-grounding
Source: https://github.com/NVIDIA-TAO/tao-skill-bank/tree/main/skills/data/tao-generate-image-grounding
Command: npx skills add https://github.com/NVIDIA-TAO/tao-skill-bank --skill tao-generate-image-grounding

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nvcr.io/nvidia/tao/tao-toolkit:6.26.3-pyt, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill automates the process of grounding captions to pixel-space bounding boxes, generating phrase-grounded annotations, and auto-labeling images for grounding. It uses Vision-Language Models (VLMs) to extract referring expressions from image-caption pairs.

Core Features & Use Cases

  • Image Grounding: Extracts referring expressions from (image, caption) pairs and grounds them to pixel-space bounding boxes.
  • Phrase Grounding: Generates phrase-grounded annotations for each expression.
  • Auto-labeling: Automatically labels images for grounding.
  • Use Case: Ideal for generating phrase-grounded training data for referring-expression and grounding models.

Quick Start

Use the tao-generate-image-grounding skill to extract referring expressions and bounding boxes from the image 'cat.jpg' and its caption 'A cute cat'.

Frequently Asked Questions about tao-generate-image-grounding

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate image grounding to generate pixel-space bounding boxes from captions?

Image grounding extracts referring expressions from image-caption pairs and grounds them to pixel-space bounding boxes. Vision-Language Models automate this extraction, generating phrase-grounded annotations for each expression.

Can I use vLLM or Gemini for auto-labeling phrase grounding annotations?

Yes, auto-labeling for phrase grounding supports Gemini, NIM, vLLM, and other OpenAI-compatible endpoints. These Vision-Language Models automate the extraction of referring expressions from image-caption pairs.

What is phrase grounding and how does it work with Vision-Language Models?

Phrase grounding generates phrase-grounded annotations for referring expressions within captions. Vision-Language Models process image-caption pairs to extract these expressions and ground them to corresponding pixel-space bounding boxes.

Do I need Python and specific image files to generate referring expression annotations?

Yes, generating referring expression annotations requires Python and access to an image file alongside its corresponding caption file. These inputs are processed through Vision-Language Model endpoints to produce grounded annotations.

What's the best way to create phrase-grounded training data for grounding models?

Automated phrase-grounded training data is generated by extracting referring expressions and bounding boxes from image-caption pairs. This approach is ideal for producing training data for referring-expression and grounding models.

Does this image grounding approach work with OpenAI-compatible endpoints?

Yes, the image grounding mechanism supports Gemini, NIM, vLLM, and other OpenAI-compatible endpoints. This allows flexible integration with various Vision-Language Models to extract referring expressions.