python-performance-optimization

Use cProfile to measure function-level execution time, then sort results by cumulative time to find hot paths. For line-by-line detail, use line_profiler, and for running production processes, attach py-spy to generate flamegraphs without code changes.

Replace lists with generators for large datasets, add __slots__ to classes with many instances, and use weakref-based caches so objects can be garbage collected. Track allocations with memory_profiler and tracemalloc to confirm reductions.

cProfile is built-in and gives deterministic function-level stats during development. py-spy is a sampling profiler that attaches to running processes with low overhead, making it suitable for profiling production systems without restarting them.

Use multiprocessing for CPU-bound tasks like heavy calculations, since it bypasses the GIL with separate processes. Use asyncio with libraries like aiohttp for I/O-bound work such as concurrent HTTP requests, where tasks wait on network or disk.

Optimizing without profiling often targets the wrong code path. Measure first with cProfile or timeit, focus on hot loops and algorithmic complexity, and benchmark before and after each change to verify actual improvement.