portal-data-connectors

Wrap the synchronous Polars scan in asyncio.to_thread or run_in_executor so the IO runs on a threadpool. Never call pl.scan_parquet directly inside an async handler, since Polars IO is synchronous and would block the event loop.

Store a (timestamp, value) tuple per key in a dict and compare time.monotonic() against a configurable TTL, such as SILO_CACHE_TTL_SECONDS defaulting to 300. Provide an invalidate function that clears one key or the whole cache.

Use pl.scan_parquet, which returns a LazyFrame and defers materialization so downstream query compilation decides what to load. pl.read_parquet loads everything into memory eagerly, which wastes resources when only a subset is needed.

Raise typed exceptions like SiloUnavailableError carrying the silo name, then catch them per-silo in the service layer and return an HTTP 503 for that silo only. Other silos continue responding normally, achieving graceful degradation.

Use pytest-asyncio with pytestmark = pytest.mark.asyncio, monkeypatch the sync scan function to record calls, and call the extractor twice. Assert the scan ran exactly once, proving the second call was served from cache.