What problem does it solve?
SENTINEL proactively monitors RRQ's infrastructure, detects failures before they impact users, and attempts automated recovery, ensuring system stability during autonomous operations.
Core Features & Use Cases
- Automated Monitoring: Continuously watches Lambda workers, EC2 instances, Inngest workflows, DynamoDB, ElevenLabs quotas, and S3 buckets for failures.
- LLM-Guided Recovery: Employs Haiku to generate and execute specific retry strategies for detected issues.
- Intelligent Escalation: Escalates to human-readable alerts (SMS, in-app notifications) only after automated retries fail and only in Full Autopilot Mode.
- Use Case: If an
audio-gen Lambda worker fails due to an ElevenLabs API quota issue, SENTINEL will detect the failure, attempt to rotate to a new account, and if successful, resolve the issue without human intervention. If retries fail, it will send an SMS and in-app notification.
Quick Start
Ensure SENTINEL is active by setting channelMode to AUTOPILOT and monitor infrastructure health.