skills.rest
MLT-OSS
MLT-OSSOFFICIAL
Explore all skills by this author →
WebsiteGitHub

llava

Run vision-language inference and fine-tuning with LLaVA models for image understanding.

14|5|Updated Apr 9, 2026
Search Tags:#fine-tuning#multimodal#clip#vision-language#vqa#llava#image captioning
One-click install
npx skills add https://github.com/MLT-OSS/hermes-agent-go --skill llava-mlt-oss
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llava
Source: https://github.com/MLT-OSS/hermes-agent-go/tree/main/optional-skills/mlops/llava
Command: npx skills add https://github.com/MLT-OSS/hermes-agent-go --skill llava-mlt-oss

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, torch, pillow, and includes references (resource) components.

What problem does it solve? Building conversational image understanding requires combining vision encoders with language models, which involves complex setup, high VRAM demands, and non-trivial training pipelines. This Skill provides ready-to-use instructions for running LLaVA inference, multi-turn image chat, and custom fine-tuning. ## Core Features & Use Cases - Vision-Language Inference: Load LLaVA models (7B, 13B, 34B) for image captioning, visual question answering, and scene understanding with Python or CLI. - Multi-Turn Image Chat: Maintain conversation context across multiple questions about the same image, with Gradio web UI support. - Custom Fine-Tuning: Train on your own instruction data using full fine-tuning with DeepSpeed or memory-efficient LoRA. - Use Case: A team building a document analysis chatbot loads llava-v1.5-7b with 4-bit quantization on a single GPU, then fine-tunes it on their domain-specific document images using LoRA. ## Quick Start Use the llava skill to load the llava-v1.5-7b model and answer questions about an image file.

Frequently Asked Questions about llava

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run LLaVA for visual question answering in Python?▼

Load a pretrained model with llava.model.builder.load_pretrained_model, process the image with the image processor, build a conversation prompt using conv_templates, and call model.generate with the image tensor. The llava-v1.5-7b checkpoint from Hugging Face works out of the box.

What is the difference between LLaVA 7B, 13B, and 34B models?▼

The 7B model needs about 14 GB VRAM and offers good quality, the 13B needs about 28 GB with better results, and the 34B needs about 70 GB for the best quality. With 4-bit quantization, VRAM drops roughly 4x for each size.

Can LLaVA run on a GPU with limited VRAM?▼

Yes, enable 4-bit quantization by passing load_4bit=True to load_pretrained_model, which reduces VRAM usage by about 4x. The 7B model then fits in roughly 4 GB, though a GPU is still required since CPU inference is extremely slow.

How do I fine-tune LLaVA on custom image data?▼

Format your data as JSON with image paths and human/gpt conversation pairs, then run the fine-tune script with DeepSpeed. For lower memory usage, apply LoRA with rank 8 targeting q_proj and v_proj, which lets a 7B model train on a single 40GB A100.

What are the limitations of LLaVA for image understanding?▼

LLaVA can hallucinate objects not present in images, struggles with precise spatial reasoning and counting many objects, and has difficulty reading small text. It also requires a capable GPU and runs slower than simpler models like CLIP.

Related Skills

Explore More Software Engineering Skills→
heysuhasheysuhas
2

llava

Enable multimodal visual instruction tuning and image-based conversational analysis.

Community
Advanced
hhhi21ghhhi21g

llama-factory

Fine-tune large language models through LLaMA-Factory's web interface.

Community
Advanced
diegosouzapwdiegosouzapw
59.5k

cli-eval

Create and run LLM evaluation suites, scorecards, and model comparisons from the OmniRoute CLI.

Community
Intermediate
qcmuuqcmuu

clip

Embed images and text for zero-shot classification and similarity scoring.

Community
Basic