skills.rest
tamagusko
tamaguskoCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

llava

Run vision-language conversations and visual question answering with LLaVA models.

Updated Oct 7, 2022
Search Tags:#multimodal#clip#vision-language#vqa#visual question answering#llava#image chat
One-click install
npx skills add https://github.com/tamagusko/linux-cfg --skill llava-tamagusko
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llava
Source: https://github.com/tamagusko/linux-cfg/tree/main/dotfiles/claude/skills/llava
Command: npx skills add https://github.com/tamagusko/linux-cfg --skill llava-tamagusko

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, torch, pillow, and includes references (resource) components.

What problem does it solve? Building image-understanding chatbots and visual question answering systems requires combining vision encoders with language models, which involves complex setup, model loading, and conversation management that this Skill streamlines. ## Core Features & Use Cases - Multi-turn Image Conversations: Maintain conversational context across multiple questions about the same image using LLaVA's conversation templates. - Flexible Model Sizes: Choose from 7B, 13B, or 34B parameter models with 4-bit or 8-bit quantization to fit available GPU VRAM. - Multiple Interfaces: Run inference via Python API, command-line interface, or Gradio web UI for interactive demos. - Use Case: A researcher building a document analysis tool can load LLaVA-v1.5-7B with 4-bit quantization on a single GPU, then ask multi-turn questions like "What is the main topic?" followed by "Summarize the key findings" about document images. ## Quick Start Ask the assistant to load the LLaVA-v1.5-7B model and answer questions about an image file you provide.

Frequently Asked Questions about llava

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run LLaVA for visual question answering in Python?▼

Load a pretrained model with llava.model.builder.load_pretrained_model, process the image with the image processor, build a conversation using conv_templates, and call model.generate with the tokenized prompt and image tensor. The response is decoded from the output token IDs.

What is the difference between LLaVA 7B, 13B, and 34B models?▼

The 7B model needs about 14 GB VRAM and offers good quality, the 13B needs about 28 GB with better quality, and the 34B needs about 70 GB for the best quality. Larger models produce more accurate answers but run slower and require more GPU memory.

Can LLaVA run on a GPU with limited VRAM?▼

Yes, LLaVA supports 4-bit and 8-bit quantization via the load_4bit or load_8bit parameters, reducing VRAM usage by roughly 4x or 2x. A 7B model in 4-bit mode needs only about 4 GB of VRAM.

LLaVA vs GPT-4V for image understanding tasks?▼

GPT-4V offers the highest quality through a hosted API, while LLaVA is open-source under Apache 2.0 and runs locally on your own GPU. Choose LLaVA when you need self-hosting, fine-tuning on custom data, or no per-request API costs.

What are the limitations of LLaVA for image analysis?▼

LLaVA can hallucinate objects not present in images, struggles with precise spatial reasoning and small text, and counts objects imprecisely. It also requires a GPU for practical inference speeds, as CPU inference is extremely slow.

How do I fine-tune LLaVA on custom image data?▼

Prepare instruction data as JSON with image paths and human/gpt conversation pairs, then run the fine-tuning script with DeepSpeed. LoRA fine-tuning reduces memory requirements significantly, allowing a 7B model to train on a single 40 GB GPU.

Related Skills

Explore More Data & Analytics Skills→
ricablericable

ray-data

Streams distributed, batched ML data preprocessing across clusters for PyTorch and TensorFlow workflows.

Community
Advanced
badhopebadhope

openai

Integrate OpenAI chat, embedding, image, and transcription APIs in Python or TypeScript.

Community
Intermediate
BoHeFanBoHeFan
42

pdf

Fill PDF form fields and extract text, tables, and metadata.

Community
Intermediate
arsityarsity

blip-2-vision-language

Generate captions, answer visual questions, and retrieve images with BLIP-2.

Community
Advanced