Run batch and interactive LLM inference with a vLLM backend.
npx skills add https://github.com/atrawog/overthink-plugins --skill inference
Please help me install this Agent Skill. Skill: inference Source: https://github.com/atrawog/overthink-plugins/tree/main/overthink-jupyter/skills/inference Command: npx skills add https://github.com/atrawog/overthink-plugins --skill inference
This skill enables fast and memory-efficient inference for large language models by leveraging a vLLM backend and unsloth, reducing latency and resource usage in both interactive and batch scenarios.
Run a sample inference by loading a pre-quantized thinking model and enabling fast_inference to observe accelerated generation.
High-intent search queries and answers about installing and using this skill.
Batch LLM inference with vLLM supports memory-managed generation for multiple prompts, utilizing unsloth and PyTorch to maintain throughput across large processing workloads.
Yes, vLLM inference works with thinking models by applying specific prompt templating and output parsing logic to properly manage and extract the generated reasoning content.
vLLM inference reduces latency and resource usage in both interactive and batch scenarios, distinguishing itself from standard generation by optimizing memory management and accelerating output.
Memory-efficient inference with vLLM requires compatible hardware to operate effectively, meaning deployments are constrained by available GPU resources and proper PyTorch environment configurations.
jayll1303Fine-tune LLMs with Unsloth kernels supporting SFT, DPO, GRPO, and GGUF export.
OpenHandsRead, modify, execute, and convert Jupyter notebooks to HTML, Python, or Markdown.
JeffallanAutomate fine-tuning workflows for large language models using LoRA/QLoRA and PEFT methods.
huggingfaceEvaluate Hugging Face Hub models locally with inspect-ai and lighteval.
