plugin-inference-middleware

To build custom inference middleware, you implement the NemoInferenceMiddleware base class to intercept and transform request and response traffic. This allows you to modify payloads, apply custom guardrails, and manage advanced model routing within the NeMo Platform.

Yes, you can redact PII from chat payloads by implementing the process_request hook in your custom middleware. This intercepts incoming traffic, mutates the request payload to strip sensitive information, and forwards the sanitized data to the model.

You can configure inference middleware using inline configurations or externalized config entities managed through a CRUD API. This externalized approach supports versioned middleware logic, enabling shared configuration management across the NeMo Platform entity store.

Yes, NeMo inference middleware supports short-circuiting requests with immediate responses. By implementing the process_request and process_response lifecycle hooks, you can intercept traffic, validate payloads, and return custom responses without executing the standard model inference.

Yes, pydantic is a required dependency for implementing custom inference middleware on the NeMo Platform. It provides the data validation and settings management needed to support standardized lifecycle hooks and payload mutation within the Inference Gateway.

You append custom guardrail metadata by implementing the process_response hook in your middleware class. This hook intercepts the model's outgoing response traffic, allowing you to mutate the payload and inject additional guardrail metadata before it returns to the client.