blip-2-vision-language

Load Blip2Processor and Blip2ForConditionalGeneration from HuggingFace Transformers with the Salesforce/blip2-opt-2.7b checkpoint, pass a PIL RGB image through the processor, and call model.generate with max_new_tokens set. Decode the output token IDs with processor.batch_decode to get the caption string.

BLIP-2 excels at zero-shot captioning and VQA with low training cost since only the Q-Former is trained. InstructBLIP is its successor with better instruction following, while LLaVA targets instruction-following multimodal chat. For production chat, proprietary models like GPT-4V are alternatives.

blip2-opt-2.7b needs about 8GB in FP16, blip2-opt-6.7b about 16GB, and blip2-flan-t5-xxl about 26GB. Using INT8 or INT4 quantization via BitsAndBytesConfig reduces these to roughly 3-8GB, letting larger variants fit on consumer GPUs.

The model exceeds available VRAM for the chosen precision and batch size. Fix it by enabling 8-bit or 4-bit quantization, switching to a smaller variant like blip2-opt-2.7b, processing images sequentially with torch.cuda.empty_cache between calls, or offloading layers to CPU.

Yes, LoRA fine-tuning with the peft library is the recommended approach, targeting attention projection layers while keeping the frozen vision encoder and LLM unchanged. You can also train only the Q-Former by freezing all parameters whose names do not contain qformer.

Hallucinations come from high-temperature sampling and vague prompts. Lower the temperature, use beam search with do_sample disabled, ask specific questions like "Is there a person in this image?", and prefer the FlanT5 variants which follow instructions more reliably.