Running Batch Jobs on GCP via dsub

Batch-execute containerized viral genomics workloads on Google Cloud with dsub.

198|67|Updated Sep 25, 2014
One-click install
npx skills add https://github.com/broadinstitute/viral-ngs --skill running-batch-jobs-on-gcp-via-dsub
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: Running Batch Jobs on GCP via dsub
Source: https://github.com/broadinstitute/viral-ngs/tree/main/.agents/skills/dsub-batch-jobs
Command: npx skills add https://github.com/broadinstitute/viral-ngs --skill running-batch-jobs-on-gcp-via-dsub

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you execute compute-heavy viral NGS tasks on Google Cloud in a reliable, scalable way when running locally is too slow, too memory-intensive, or requires specialized Docker images.

Core Features & Use Cases

  • Parallel batch execution with dsub: Runs many independent jobs from a single task TSV to maximize throughput.
  • Dockerized tool execution: Uses a specified container image so tool versions and dependencies stay consistent.
  • GCS-first data flow: Downloads declared inputs from GCS and uploads outputs/logs back to GCS for auditing and debugging.
  • Common viral analysis scenarios: Useful for memory-heavy steps like VADR, BLAST, and genome assembly, especially when processing dozens of sequences or assemblies.

Quick Start

Run dsub with the GCS-based provider settings, a Docker image, a machine type, a script, and a tasks TSV so each TSV row launches an independent containerized job and writes logs to your GCS logging path.

Frequently Asked Questions about Running Batch Jobs on GCP via dsub

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run parallel viral genomics jobs on Google Cloud with dsub?

To run parallel viral genomics jobs on Google Cloud with dsub, you provide a Docker image, a shell script, a TSV task file defining input and output mappings, and a GCS logging path. Each TSV row launches an independent containerized job on Google Batch.

Can I use dsub to batch process large NGS datasets on GCP?

Yes, you can use dsub to batch process large NGS datasets on GCP. It is designed for high-RAM viral analysis tools like VADR and BLAST, downloading inputs from GCS and uploading outputs back to GCS for scalable parallel execution.

What do I need to set up before running dsub batch jobs on Google Cloud?

Before running dsub batch jobs on Google Cloud, you need a valid GCP project and region, authenticated gcloud access, a Docker image, a shell script, a TSV task file defining environment and input mappings, and a GCS logging location.

What is the best way to execute memory-heavy genome assembly tasks across many sequences?

The best way to execute memory-heavy genome assembly tasks across many sequences is using dsub on Google Cloud. It applies containerized Docker images to process dozens of independent input and output pairs in parallel using Google Batch.

Does dsub support Dockerized bioinformatics tools on Google Cloud Batch?

Yes, dsub supports Dockerized bioinformatics tools on Google Cloud Batch. It uses a specified container image so tool versions and dependencies stay consistent across all parallel jobs launched from your task TSV file.

When should I not use dsub for parallel batch processing on GCP?

You should not use dsub for parallel batch processing on GCP if your tasks are not independent or if they do not require high-RAM compute. dsub is built for large task sets where each input and output pair can be processed independently in parallel.