seldon-core

ML model serving on Kubernetes with canary rollouts is handled using SeldonDeployment CRD definitions. You configure traffic splitting for safe, gradual rollouts, enabling shadow deployments and validation before routing full inference traffic to the new model version.

Custom Python model servers are supported for complex inference pipelines on Kubernetes. They handle preprocessing, postprocessing, and multi-step inference logic, allowing you to wrap custom logic around your models before serving them across REST or gRPC endpoints.

Istio integration is supported for advanced traffic management across REST and gRPC endpoints. It provides mTLS, telemetry, and advanced routing capabilities for scalable model serving, ensuring observable and secure inference traffic on Kubernetes.

Autoscaling for model serving on Kubernetes is configured through SeldonDeployment CRD definitions. It works with REST and gRPC endpoints to automatically scale inference workloads, managing traffic and observability for scalable production deployments.

SeldonDeployment CRD definitions are required to expose a model via REST on Kubernetes. They define the deployment configuration, allowing you to validate routing, autoscaling, and traffic management for your inference endpoints.