rasa-configuring-model-groups

Configure multiple LLM deployments in Rasa endpoints.yml by defining model_groups with routing strategies such as least-busy or latency-based, enabling efficient distribution of requests across available model instances.

Model groups in Rasa are endpoints.yml configurations that organize LLM and embedding providers. You need them when integrating multiple language models to enable flexible routing, failover mechanisms, and response caching for robust conversational agents.

Yes, you can set up failover and caching for self-hosted LLM providers in Rasa. Configure model_groups in endpoints.yml to define failover mechanisms and implement response caching, ensuring robust operation during outages or high traffic.

Rasa supports configuring Azure OpenAI and other external embedding providers. Set up various LLM and embedding providers within endpoints.yml using model_groups to manage routing and integration seamlessly.

The best way to route LLM requests based on latency in a Rasa chatbot is configuring model_groups in endpoints.yml with a latency-based routing strategy. This directs traffic to the fastest responding deployment, optimizing agent performance.