skills.rest
vipshop
vipshopOFFICIAL
Explore all skills by this author →
WebsiteGitHub

cache-dit-model-integration

Integrates new DiT models into cache-dit with cache, parallelism, and CLI support.

1.3k|80|Updated Jun 12, 2025
Search Tags:#model integration#diffusers#inference optimization#cache-dit#context parallelism#Tensor Parallelism#diffusion transformer
One-click install
npx skills add https://github.com/vipshop/cache-dit --skill cache-dit-model-integration
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: cache-dit-model-integration
Source: https://github.com/vipshop/cache-dit/tree/main/.github/skills/cache-dit-model-integration
Command: npx skills add https://github.com/vipshop/cache-dit --skill cache-dit-model-integration

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Adding support for a new diffusion transformer (DiT) model in cache-dit requires coordinated changes across caching adapters, context parallelism, tensor parallelism, text encoder and VAE parallelism, CLI registration, and correctness testing. This Skill provides a gated, phase-by-phase workflow that prevents silent output corruption and wasted implementation effort.

Core Features & Use Cases

  • Cache Integration: Guides BlockAdapter creation, ForwardPattern selection (Pattern_0 through Pattern_5), and PatchFunctor implementation for structural mismatches.
  • Parallelism Planning: Provides decision charts and templates for Context Parallelism (hook-based and hybrid), Tensor Parallelism (ColwiseParallel/RowwiseParallel, GQA handling), Text Encoder Parallelism, and VAE Parallelism.
  • Verification Workflow: Enforces PSNR and SSIM correctness checks against single-GPU baselines at every phase, with documented pitfalls like attention mask reordering and shard_div_attr bugs.
  • Use Case: When adding support for a new model like Krea-2-Turbo, follow the TODO checklist to implement the BlockAdapter, register the generate CLI entry, add CP and TP planners, and validate each stage with PSNR > 35 dB and SSIM > 0.90 before proceeding.

Quick Start

Ask the assistant to integrate a new DiT model into cache-dit, providing the local model path and the pipeline and transformer class names with their source file paths.

Frequently Asked Questions about cache-dit-model-integration

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add a new DiT model to cache-dit?▼

Provide the local model path plus the pipeline and transformer class names with source file paths, then follow the phased workflow: Cache BlockAdapter first, then Context Parallelism, Tensor Parallelism, TE-P, VAE-P, and CLI registration. Each phase must pass PSNR and SSIM checks against a single-GPU baseline before proceeding.

How do I choose the right ForwardPattern for a transformer block?▼

Read the block's forward() signature in the diffusers source: check whether it takes only hidden_states or also encoder_hidden_states, and whether it returns one tensor or two and in what order. Match these against the six ForwardPattern contracts, such as Pattern_1 for Flux-style dual-stream blocks.

Should I use hook-based or patch-based Context Parallelism?▼

Use hook-based CP whenever sequence tensors are plain tensors passed to nn.Module forward calls, and hybrid CP with a minimal method patch when one or two tensors like RoPE embeddings are unreachable by hooks. Pure patch-based CP that rewrites the entire forward() is explicitly rejected.

Why is my CP output corrupted in the top-left corner?▼

This is the classic attention mask reordering bug: Ulysses all-to-all reorders the sequence into rank-concatenated order, so a position-indexed attention_mask no longer aligns. The fix is to permute the mask along the key dimension, or both query and key dimensions for 2D masks.

Why is PSNR alone not enough to verify cache-dit correctness?▼

A garbled image can still score PSNR above 25 dB, so PSNR alone cannot detect corrupted output. Always compute both PSNR and SSIM, requiring PSNR above 30-35 dB and SSIM above 0.90 depending on the phase, plus visual inspection.

Can I modify diffusers library code when integrating a model?▼

No, diffusers is a third-party dependency and must not be altered. All patches such as forward() monkey-patching or attention processor changes must be written inside the cache-dit repository, typically as PatchFunctor classes or planner modules.

Related Skills

Explore More Software Engineering Skills→
vllm-projectvllm-project
6.0k

add-diffusion-model

Add diffusion models to vLLM-Omni with Cache-DiT acceleration and multi-GPU parallelism.

Official
Advanced
sayalinvidiasayalinvidia

nemo-mbridge-perf-parallelism-strategies

Select and tune Megatron Bridge parallelism configurations for model and hardware.

Community
Advanced
spacecat2002spacecat2002

sglang-diffusion-performance

Select SGLang Diffusion configuration flags for generation speed and GPU memory usage.

Community
Advanced
nexu-ionexu-io
93.2k

community-import-smoke-test

Validate OpenDesign plugin import flows across local folders, zip archives, and remote repositories.

Official
Basic