machine-learning-engineer

Convert the model to ONNX for faster inference, wrap it in a FastAPI or gRPC endpoint, and deploy it to Kubernetes with a horizontal pod autoscaler. Add request batching, health checks, and Prometheus metrics to meet latency and throughput targets.

Apply quantization from FP32 to FP16 or INT8, prune unnecessary weights, and export to ONNX or TensorRT. These techniques typically cut model size by 4x and significantly reduce inference time with minimal accuracy loss.

Yes, models can be quantized to INT8 and run with CoreML on iOS or TFLite on Android for on-device inference. This enables offline capability, lower latency, and privacy compliance since data never leaves the device.

Real-time systems serve predictions per request with strict latency targets, while batch systems process large datasets on a schedule with job orchestration and parallel processing. Choose based on whether predictions are needed immediately or can be precomputed.

Model degradation usually comes from data drift, where input distributions shift away from training data. Monitor prediction distributions, error rates, and input statistics, and set alerts to trigger retraining when drift is detected.