mesh-job-orchestrator

You orchestrate distributed compute jobs by decomposing large tasks into manageable work units using strategies like MapReduce, Pipeline, ScatterGather, or LayerSplit. This enables scalable parallel execution, dynamic scheduling, and result aggregation across a compute mesh.

To track progress and aggregate results for parallel workflows, the system monitors work units dynamically, retries failed units, and aggregates outputs to provide end-to-end visibility. You can retrieve progress reports and statistics via a typed API.

You need MapReduce or ScatterGather strategies when processing large-scale data or executing complex workflows that require scalable, multi-worker execution. These decomposition strategies allow heterogeneous environments to dynamically schedule and process work units in parallel.

Yes, you can cancel a distributed compute job after it has started. The typed API supports actions for cancelling active jobs, as well as creating, retrieving, listing, and reporting progress on work units to maintain full control over execution.

To handle failed work units in a multi-worker execution environment, the orchestration system automatically retries failed units. It tracks completion and failure states through the typed API, ensuring robust execution and accurate result aggregation for complex workflows.