retraining-worker

Automate model retraining and promotion by using a Kafka-triggered worker that trains a challenger model and shadow tests it against current performance before promotion. This workflow rebuilds anomaly detection models from recent data without manual intervention.

Yes, you can use Celery for non-blocking model training jobs triggered by Kafka. The worker uses a subprocess worker pattern with task retries to safely dispatch and execute model training jobs in the background.

Shadow testing evaluates the new model against the current one to ensure it performs at least as well. When the challenger model wins the shadow test, it is promoted and written to Redis for version control.

Limitations include missing scored events, bootstrap promotion behavior, and process-level metric limitations. The system relies on Redis for model versioning and TimescaleDB for data access, which can affect operational health monitoring.

Handle retry logic for failed model retraining tasks using the Celery execution model. The worker implements task retries and non-blocking dispatch to manage subprocess worker failures safely during the anomaly detection training process.

The best way to trigger model retraining upon drift detection is consuming Kafka events. The retraining worker reacts to drift signals, automatically rebuilding the anomaly detection model from recent data and promoting the challenger.