parallel-python

Size Python multiprocessing workers by matching the process count directly to the SLURM_CPUS_PER_TASK environment variable. This prevents CPU oversubscription and underutilization on the HPC cluster.

Use Slurm arrays instead of Python multiprocessing when your tasks are embarrassingly parallel. Slurm arrays handle independent workloads more efficiently by distributing jobs across the cluster natively.

The best way to manage BLAS threading is maintaining a single main parallelism layer. Control BLAS thread counts explicitly to prevent nested parallelism from causing performance degradation and oversubscription.

Choose a multiprocessing start method based on workload characteristics and apply dynamic load balancing to handle uneven tasks. This ensures stable performance and prevents workers from idling.

Mixing threading and multiprocessing is discouraged because it creates nested parallelism. You should select one main parallelism layer—either threads for I/O-bound tasks or processes for CPU-bound tasks—to maintain stable performance.

Python threading does not work well for CPU-bound tasks due to the Global Interpreter Lock. For CPU-bound workloads on HPC clusters, use multiprocessing or Slurm arrays to achieve true parallelism.