together-batch-inference

You can process bulk inference requests asynchronously by uploading a JSONL file to the Together AI Batch API, creating a job, and downloading the results once polling shows completion. This workflow handles up to 50K requests per batch.

The best way to lower costs for large-scale synthetic data generation is using batch inference. By aggregating offline inference requests into bulk jobs, you avoid the premium of real-time processing and optimize workload expenses.

Together AI batch inference supports input files up to 100MB in size and allows a maximum of 50,000 requests per batch. These limits ensure scalable asynchronous processing without overwhelming the API.

Yes, you can use JSONL files for bulk data classification and offline inference. The workflow accepts JSONL input format to aggregate, submit, and retrieve classification results asynchronously.

You monitor an asynchronous batch inference job through status polling. After submitting the JSONL input file, the workflow periodically checks the job status until processing completes and results are ready for download.

You should use batch processing for large workloads because sending individual inference requests is slow and costly. Batch processing aggregates requests for asynchronous execution, optimizing both processing time and overall expenses.