skills.rest
xu1713
xu1713COMMUNITY
Explore all skills by this author →
GitHub

llava

Run vision-language conversations and visual question answering with LLaVA models.

Updated Jun 5, 2026
Search Tags:#fine-tuning#multimodal#vision-language#vqa#visual question answering#llava#image chat
One-click install
npx skills add https://github.com/xu1713/openhorse --skill llava-xu1713
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llava
Source: https://github.com/xu1713/openhorse/tree/main/openhorse/openhorse/optional-skills/mlops/llava
Command: npx skills add https://github.com/xu1713/openhorse --skill llava-xu1713

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, torch, pillow, and includes references (resource) components.

What problem does it solve? Building image-understanding chatbots typically requires expensive proprietary APIs like GPT-4V. This Skill lets you run open-source vision-language models locally for image captioning, visual question answering, and multi-turn image conversations. ## Core Features & Use Cases - Visual Question Answering: Ask natural-language questions about images and receive detailed answers using CLIP vision encoders combined with Vicuna/LLaMA language models. - Multi-Turn Image Chat: Maintain conversational context across multiple questions about the same image, with CLI, Python API, and Gradio web UI interfaces. - Custom Fine-Tuning: Train on your own instruction data with full fine-tuning or memory-efficient LoRA, plus 4-bit/8-bit quantization to reduce VRAM requirements. - Use Case: Deploy a self-hosted document understanding assistant that answers questions about scanned forms and charts without sending data to external APIs. ## Quick Start Ask the assistant to load the llava-v1.5-7b model and describe what is in a given image file.

Frequently Asked Questions about llava

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run visual question answering with LLaVA in Python?▼

Load a pretrained model with llava.model.builder.load_pretrained_model, process the image with the image processor, build a conversation prompt using conv_templates, and call model.generate with the image tensor. The response is decoded from the output token IDs.

What is the difference between LLaVA and GPT-4V for image understanding?▼

LLaVA is an open-source Apache 2.0 model you run locally on your own GPU, while GPT-4V is a higher-quality API-based service. LLaVA targets GPT-4V-level capabilities and suits self-hosted deployments where data privacy matters.

How much VRAM does LLaVA need for inference?▼

The 7B model needs about 14 GB in FP16 or 4 GB with 4-bit quantization. The 13B model needs roughly 28 GB FP16 or 8 GB quantized, and the 34B model needs about 70 GB FP16 or 18 GB quantized.

Can I fine-tune LLaVA on my own image dataset?▼

Yes, prepare instruction data as JSON with image paths and human/gpt conversation pairs, then run the fine-tuning script with DeepSpeed. LoRA fine-tuning reduces memory needs to a single A100 for the 7B model.

What are the limitations of LLaVA for image analysis?▼

LLaVA can hallucinate objects not present in images, struggles with precise spatial reasoning and small text, and counts objects imprecisely. It also requires a capable GPU since CPU inference is extremely slow.

Related Skills

Explore More Software Engineering Skills→
aurabxaurabx
6

Medical Imaging Pipelines

Convert DICOM medical images to NIfTI, PNG, JPEG, and HDF5 formats.

Official
Advanced
X-School-AcademyX-School-Academy
19

media-image-analysis

Analyze images with AI vision models to answer questions about content.

Official
Basic
tassiovaletassiovale
13

liteparse

Extract text and bounding boxes from PDFs, Office files, and images.

Community
Intermediate
gagan114662gagan114662

clip

Classifies images into text-defined categories using OpenAI's CLIP model.

Community
Intermediate