skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

llava

Combine CLIP vision encoders with large language models for visual question answering.

2|Updated Apr 12, 2026
Search Tags:#multimodal#quantization#clip#vision-language#vqa#llava#image-chat
One-click install
npx skills add https://github.com/Clay-HHK/claude-config --skill llava-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llava
Source: https://github.com/Clay-HHK/claude-config/tree/main/skills/AI-research-SKILLs/18-multimodal/llava
Command: npx skills add https://github.com/Clay-HHK/claude-config --skill llava-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill packages an open-source vision-language assistant that enables conversational image understanding and visual question answering, removing the need to build multimodal pipelines from scratch and accelerating prototype and research workflows.

Core Features & Use Cases

  • Conversational Image Chat: Supports multi-turn dialogues grounded in images for chatbots, assistants, and interactive demos.
  • Visual Question Answering (VQA) & Captioning: Answer scene questions, count objects, describe images, and generate detailed captions.
  • Training & Fine-tuning Guidance: Includes instructions for feature alignment, visual instruction tuning, LoRA, and DeepSpeed-based training workflows for custom datasets.
  • Deployment Flexibility: Guidance for running different model sizes (7B–34B), GPU inference, and optional 4-bit/8-bit quantization to reduce VRAM.

Quick Start

Ask llava to analyze the provided image and answer what is visible in the scene.

Frequently Asked Questions about llava

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a conversational image chatbot that supports visual question answering?▼

Conversational image chat and visual question answering (VQA) are achieved by combining CLIP vision encoders with large language models for multi-turn dialogues grounded in images.

Can I fine-tune a vision-language model using LoRA and DeepSpeed for custom datasets?▼

Yes, visual instruction tuning and feature alignment can be fine-tuned using LoRA and DeepSpeed workflows to adapt the vision-language model to custom datasets.

How do I run large multimodal models for GPU inference with limited VRAM?▼

GPU inference with limited VRAM is supported by applying optional 4-bit or 8-bit quantization to model sizes ranging from 7B to 34B.

What is the best way to perform image captioning and document image understanding?▼

Image captioning and document image understanding are performed by combining vision encoders with large language models to describe scenes, count objects, and generate detailed captions.

Does this multimodal pipeline support counting objects and describing scenes in images?▼

Yes, the multimodal pipeline supports visual question answering to count objects, describe scenes, and generate detailed captions from provided images.

Related Skills

Explore More Education & Research Skills→
thisismynewfmail-uithisismynewfmail-ui

llava

Enable conversational image understanding and visual question answering for multimodal inputs.

Community
Advanced
GarrettRoiGarrettRoi

llava

Combine CLIP vision encoders with Vicuna and LLaMA for visual question answering.

Community
Advanced
tianhao909tianhao909
1

llava

Combine large language models with vision encoders for image-based conversations and question answering.

Community
Advanced
levanminhduclevanminhduc

gemini-vision

Analyzes images via Google Gemini Vision for captioning, classification, VQA, and object detection with bounding boxes.

Community
Advanced