skills.rest
vivekgoquest
vivekgoquestCOMMUNITY
Explore all skills by this author →
GitHub

llava

Runs LLaVA vision-language models for image captioning, VQA, and multi-turn image dialogue.

Updated Aug 22, 2026
Search Tags:#fine-tuning#PyTorch#multimodal#vision-language#vqa#llava#image captioning
One-click install
npx skills add https://github.com/vivekgoquest/hermes-agent-stable --skill llava-vivekgoquest
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llava
Source: https://github.com/vivekgoquest/hermes-agent-stable/tree/main/optional-skills/mlops/llava
Command: npx skills add https://github.com/vivekgoquest/hermes-agent-stable --skill llava-vivekgoquest

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, torch, pillow, and includes references (resource) components.

What problem does it solve? It enables local, open-source visual question answering and image understanding without relying on paid APIs like GPT-4V, letting you caption images, answer questions about them, and hold multi-turn image conversations on your own GPU. ## Core Features & Use Cases - Visual Question Answering: Ask natural-language questions about images using LLaVA models (7B, 13B, 34B) via Python, CLI, or a Gradio web UI. - Multi-turn Image Dialogue: Maintain conversation context across multiple questions about the same image. - Custom Fine-tuning: Train on your own instruction data with full fine-tuning or memory-efficient LoRA, including 4-bit/8-bit quantization for lower VRAM. - Use Case: A team building a document-understanding chatbot loads llava-v1.5-7b with 4-bit quantization on a single GPU and queries scanned document images through the interactive CLI. ## Quick Start Ask the agent to load the llava-v1.5-7b model and describe what is in a given image file.

Frequently Asked Questions about llava

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run LLaVA for visual question answering in Python?▼

Load a pretrained model with llava.model.builder.load_pretrained_model, process the image with the image processor, build a prompt using conv_templates, and call model.generate with the image tensor. The response is decoded from the output token IDs.

LLaVA vs GPT-4V vs BLIP-2 for image understanding?▼

LLaVA is open-source (Apache 2.0) and runs locally, targeting GPT-4V-level capabilities. GPT-4V offers the highest quality via API, while BLIP-2 is better suited for captioning-only tasks and CLIP for simple zero-shot classification.

How much VRAM does LLaVA need to run?▼

The 7B model needs about 14 GB in FP16 or 4 GB with 4-bit quantization. The 13B model needs roughly 28 GB (8 GB quantized), and the 34B model needs about 70 GB (18 GB quantized). A GPU is required since CPU inference is extremely slow.

Can I fine-tune LLaVA on my own image dataset?▼

Yes, prepare instruction data as JSON with image paths and human/gpt conversation pairs, then run the provided fine-tune script with DeepSpeed. LoRA fine-tuning reduces memory needs about 10x, letting a 7B model train on a single 40GB A100.

What are the limitations of LLaVA?▼

LLaVA can hallucinate objects not present in images, struggles with precise spatial reasoning and small text, and counts objects imprecisely. It also requires a powerful GPU and runs slower than lightweight models like CLIP.

Related Skills

Explore More Software Engineering Skills→
OwnLabAIOwnLabAI
21

histolab

Extract and preprocess tiles from whole slide images using tilers and masks.

Official
Advanced
gagan114662gagan114662

llava

Process image content for visual question answering and captioning.

Community
Advanced
matt-metiviermatt-metivier

answer-question

Ground answers on retrieved vault evidence to reduce hallucinations.

Community
Advanced
davila7davila7
30.5k

blip-2-vision-language

Generate image captions and answer visual questions using BLIP-2 vision-language models.

Community
Advanced