skills.rest
Arasz
AraszCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

llamasharp-local-inference

Runs local GGUF models in-process via LLamaSharp for text generation and judging.

2|Updated Aug 2, 2026
Search Tags:#csharp#llm#dotnet#gguf#text generation#local inference#llamasharp
One-click install
npx skills add https://github.com/Arasz/ai-raccoon --skill llamasharp-local-inference-arasz
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llamasharp-local-inference
Source: https://github.com/Arasz/ai-raccoon/tree/main/.ai-badger/skills/learned/software-development/llamasharp-local-inference
Command: npx skills add https://github.com/Arasz/ai-raccoon --skill llamasharp-local-inference-arasz

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires LLamaSharp, LLamaSharp.Backend.Cpu, and includes references (resource) components.

What problem does it solve? Running local GGUF models for text generation in this .NET repo requires navigating LLamaSharp 0.27 API pitfalls (non-disposable executors, misplaced temperature settings, central package management traps) that cause compile and build failures. ## Core Features & Use Cases - Working API Pattern: Provides a verified LLamaSharp 0.27 code snippet using StatelessExecutor, InferenceParams, and DefaultSamplingPipeline for deterministic token generation. - Pitfall Documentation: Covers CS1674 disposal errors, temperature placement, IAsyncEnumerable token streaming, Directory.Packages.props scoping, and xunit v3 namespaces. - Use Case: Use this when adding a local GGUF model as a judge or generator in a test project, including the Qwen2.5 ChatML prompt template and evidence that sub-1B models fail as promotion scorers. ## Quick Start Use the llamasharp-local-inference skill to write code that runs a local GGUF model for text generation in the test project.

Frequently Asked Questions about llamasharp-local-inference

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a local GGUF model with LLamaSharp in C#?▼

Load weights with LLamaWeights.LoadFromFile using ModelParams, create a StatelessExecutor, and call InferAsync with InferenceParams. InferAsync returns IAsyncEnumerable of token strings that you join into the final output.

How to set temperature for deterministic output in LLamaSharp?▼

Temperature is set on DefaultSamplingPipeline, not InferenceParams. Use new DefaultSamplingPipeline { Temperature = 0f } for greedy deterministic generation and assign it to InferenceParams.SamplingPipeline.

Why does using var executor fail to compile with StatelessExecutor?▼

StatelessExecutor does not implement IDisposable, so a using declaration fails with CS1674. Only LLamaWeights is disposable; dispose the weights and let the executor be garbage collected.

Why does LLamaSharp fail with NU1010 in the test project?▼

The test project resolves tests/Directory.Packages.props, a separate central package management scope that overrides the root file. Add PackageVersion entries for LLamaSharp and LLamaSharp.Backend.Cpu to tests/Directory.Packages.props.

Can a small local model like Qwen2.5-0.5B judge output quality?▼

Measured results show Qwen2.5-0.5B-Instruct scored Spearman +0.13 against a labeled fixture versus +0.40 for a mechanical scorer, and fusing it lowered accuracy. Sub-1B models are not viable as quality judges.

Related Skills

Explore More Software Engineering Skills→
AlexKonceptAlexKoncept

llama-cpp

Locates GGUF files on Hugging Face Hub and launches a local llama.cpp inference server for testing.

Community
Advanced
adrianasiam19adrianasiam19

advanced-evaluation

Implements LLM-as-judge evaluation pipelines with bias mitigation, rubrics, and calibrated scoring.

Community
Advanced
Sir-chawakornSir-chawakorn
9

llm-eval-harness

Builds an LLM evaluation harness using golden datasets and code-based scorers.

Community
Advanced
ACGSpgpACGSpgp

embedding-strategies

Select and optimize embedding models for semantic search and RAG applications.

Official
Intermediate