gem-llm-test-inference

What problem does it solve?

Verifying that a locally deployed GEM-LLM vLLM endpoint can reliably produce correct chat responses and complete tool-calling flows before you rely on it in production.

Core Features & Use Cases

• Inference smoke tests: Runs basic chat completions to confirm the model is responding as expected.
• Tool-calling verification: Exercises function/tool calls with tool_choice=auto and checks tool call outputs.
• Streaming validation: Confirms streaming behavior works and returns tokens incrementally.
• OpenAI SDK compatibility checks: Runs equivalent tests via the OpenAI client pointed at the local vLLM server.
• Logging for reporting: Stores inference transcripts in jsonl files for later review (TTFT, tokens/sec, finish_reason, tool_call success).

Quick Start

Run the skill and ask for an inference and tool-calling test by executing: ./install.sh gem-llm-test-inference.