skills.rest
alibaba
alibabaOFFICIAL
Explore all skills by this author →
GitHub

riscv-cpu-optimize

Optimizes MNN RISC-V CPU kernels and LLM inference performance on RVV and vendor matrix extensions.

16.0k|2.4k|Updated Apr 15, 2019
Search Tags:#benchmarking#llm inference#kernel optimization#riscv#GEMM#Quantization#rvv
One-click install
npx skills add https://github.com/alibaba/MNN --skill riscv-cpu-optimize
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: riscv-cpu-optimize
Source: https://github.com/alibaba/MNN/tree/main/skills/riscv-cpu-optimize
Command: npx skills add https://github.com/alibaba/MNN --skill riscv-cpu-optimize

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Optimizing LLM inference on RISC-V CPUs requires navigating layered implementations (generic CPU, standard RVV, vendor ISA like SpacemiT IME2), low-bit quantized GEMM/GEMV kernels, and remote board validation, where mistakes in pack layout, signedness, or threading silently break correctness or waste performance.

Core Features & Use Cases

  • Layered kernel design workflow: Separates generic CPU, standard RVV, and vendor-specific targets so vendor instructions, TCM usage, and shape gating stay isolated with a working RVV fallback.
  • Correctness-first methodology: Builds scalar oracles, per-block quantization checks, and A/B comparisons before touching pack, kernel, threading, or TCM pipelines.
  • Prefill and decode tuning: Applies distinct strategies for prefill (data reuse, matrix unit utilization) and decode (weight traffic, dispatch overhead, effective bandwidth roofline).
  • Remote board validation: Covers SSH-based cross-compilation, build matrices (vendor ON vs pure RVV OFF), llm_bench benchmarking, and sanitized performance reporting.
  • Use Case: When a W4B64 quantized model decodes slowly on a SpacemiT board, use this Skill to profile the hot path, implement an IME2 GEMV kernel with correct zero-point correction, and validate tokens/s gains on the target hardware.

Quick Start

Use the riscv-cpu-optimize skill to analyze and optimize the decode performance of my quantized LLM running on a RISC-V board with RVV support.

Frequently Asked Questions about riscv-cpu-optimize

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference performance on RISC-V CPUs?▼

Profile first to separate prefill and decode bottlenecks, then optimize prefill for data reuse and matrix unit utilization while optimizing decode for weight traffic and dispatch overhead. Validate all changes on the target board since ISA, VLEN, and memory bandwidth only manifest on real hardware.

How to implement low-bit quantized GEMM kernels with RVV and IME2?▼

Treat pack and kernel as one ABI, verifying tile, stride, signedness, scale, zero point, and tail handling together. For INT8 activation times INT4 weight on IME2, split the INT8 activation into signed high and unsigned low nibbles and combine two dot products.

Does MNN support SpacemiT IME2 matrix extensions on RISC-V?▼

Yes, MNN has a vendor target under source/backend/cpu/riscv/rvv/spacemit_ime2 for IME2 kernels. Vendor instructions must stay in isolated compilation targets while standard RVV remains an independently buildable fallback.

Why does adding more threads slow down RISC-V LLM decode?▼

More threads can increase contention on shared matrix units, DRAM bandwidth, and barrier costs rather than adding throughput. Run a thread-count sweep and measure sustained effective bandwidth before assuming more workers help.

Why is my quantized kernel correct on generic CPU but wrong on the vendor path?▼

Vendor-only numeric errors usually come from pack ABI mismatches, signedness confusion, or incorrect scale and zero-point correction in tail handling. Compare packed input, integer accumulator, dequantization, and epilogue stage by stage against a scalar oracle.

When should TCM be used for RISC-V kernel optimization?▼

Use TCM as an explicitly managed scratchpad only when the working set fits and copy can overlap with compute. Verify DRAM and TCM kernels produce identical results first, and fall back when capacity, runtime availability, or workload size does not justify it.

Related Skills

Explore More Software Engineering Skills→
sgl-projectsgl-project
33.0k

llm-torch-profiler-analysis

Analyzes torch.profiler traces from LLM serving frameworks into kernel, overlap, and fusion tables.

Official
Advanced
hey-konghey-kong

add-sgl-kernel

Implement and integrate CUDA/C++ kernels with tests and benchmarks.

Community
Advanced
andymaiandymai
77

writing-tests

Author and debug cross-kernel tests for the brepjs CAD JavaScript library.

Community
Advanced
qualcommqualcomm
2

sse-avx-to-neon

Translate SSE/AVX x86 intrinsics to ARM NEON equivalents.

Official
Advanced