skills.rest
Chensihakniroth
ChensihaknirothCOMMUNITY
Explore all skills by this author →
GitHub

llava

Run vision-language conversations and visual question answering with LLaVA models.

Updated Jun 7, 2026
Search Tags:#fine-tuning#multimodal#CLIP#vision-language#visual question answering#llava#image captioning
One-click install
npx skills add https://github.com/Chensihakniroth/ANAKOT-AGENT --skill llava-chensihakniroth
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llava
Source: https://github.com/Chensihakniroth/ANAKOT-AGENT/tree/main/optional-skills/mlops/llava
Command: npx skills add https://github.com/Chensihakniroth/ANAKOT-AGENT --skill llava-chensihakniroth

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, torch, pillow, and includes references (resource) components.

What problem does it solve? Building image-understanding chatbots typically requires expensive proprietary APIs like GPT-4V. This Skill lets you run open-source vision-language models locally for image captioning, visual question answering, and multi-turn image conversations without sending data to external services. ## Core Features & Use Cases - Conversational Image Analysis: Load LLaVA models (7B to 34B parameters) and chat about images across multiple turns with maintained context. - Flexible Deployment: Run via Python API, CLI, or a Gradio web interface, with 4-bit and 8-bit quantization to reduce VRAM requirements. - Custom Fine-Tuning: Train on your own instruction data using full fine-tuning or memory-efficient LoRA with DeepSpeed support. - Use Case: Build a document understanding assistant that answers questions about scanned forms, or a chatbot that describes and discusses uploaded photos. ## Quick Start Ask the agent to load the llava-v1.5-7b model and describe what is in the attached image file.

Frequently Asked Questions about llava

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run LLaVA for visual question answering in Python?▼

Load a pretrained model with llava.model.builder.load_pretrained_model, process the image with the image processor, build a conversation prompt using conv_templates, and call model.generate with the image tensor. The response is decoded from output token IDs.

What is the difference between LLaVA and CLIP for image tasks?▼

LLaVA combines a CLIP vision encoder with a language model for conversational image understanding and detailed question answering. CLIP alone only performs zero-shot classification and similarity scoring without generating descriptive text.

How much VRAM does LLaVA need to run?▼

The 7B model needs about 14 GB in FP16 or 4 GB with 4-bit quantization. The 13B model needs roughly 28 GB FP16 or 8 GB quantized, while the 34B model requires around 70 GB FP16 or 18 GB quantized.

Can I fine-tune LLaVA on my own image dataset?▼

Yes, prepare instruction data as JSON with image paths and human/gpt conversation pairs, then run the fine-tuning script with DeepSpeed. LoRA fine-tuning reduces memory needs to a single 40GB GPU for the 7B model.

What are the limitations of LLaVA for image analysis?▼

LLaVA can hallucinate objects not present in images, struggles with precise spatial reasoning and counting many objects, and has difficulty reading small text. It also requires a GPU since CPU inference is extremely slow.

Related Skills

Explore More Software Engineering Skills→
aurabxaurabx
6

Medical Imaging Pipelines

Convert DICOM medical images to NIfTI, PNG, JPEG, and HDF5 formats.

Official
Advanced
X-School-AcademyX-School-Academy
19

media-image-analysis

Analyze images with AI vision models to answer questions about content.

Official
Basic
blueskies1818blueskies1818
1

ocr-and-documents

Convert PDFs, scanned documents, and images into searchable text via OCR.

Community
Advanced
handsomelong922handsomelong922

blip-2-vision-language

Bridge frozen vision encoders with LLMs for captioning, VQA, and image-text retrieval.

Community
Advanced