huggingface-local-models

To run Hugging Face models locally, this skill finds suitable GGUF files on Hugging Face and provides llama.cpp-compatible workflows for deployment. It guides you through exact model file identification and quantization selection to launch an OpenAI-compatible inference server.

The best way to choose quantization for a local inference server is evaluating your hardware capabilities. This skill recommends suitable GGUF quants based on your specific CPU, Metal, CUDA, or ROCm environment to balance performance and resource efficiency.

Yes, you can configure llama.cpp for CUDA and ROCm hardware acceleration. This skill generates tailored llama-cli and llama-server deployment commands that specifically target your available hardware environment for optimized local model inference.

To set up an OpenAI-compatible serving configuration, this skill builds the exact llama-server commands needed for your environment. It configures the local inference server using your selected GGUF model and hardware acceleration settings.

You do not necessarily need to convert models yourself because this skill identifies existing GGUF files on Hugging Face. It provides conversion decision guidance and locates pre-quantized model files ready for direct llama.cpp deployment.

A local AI coding assistant needs a specific GGUF quant to match your hardware constraints. Selecting an inappropriate quantization can cause out-of-memory errors or suboptimal inference speed, which this skill helps avoid through hardware-aware recommendations.