python-performance-optimization

Use cProfile to profile Python code by wrapping execution with profiler.enable() and profiler.disable(), then inspect results with pstats sorted by cumulative time. For line-by-line detail, use line_profiler with kernprof, and for running production processes use py-spy.

cProfile is a deterministic profiler that instruments code and adds overhead, making it suited for development. py-spy is a sampling profiler that attaches to running processes with low overhead, making it suitable for profiling production systems and generating flamegraphs.

Use tracemalloc to take memory snapshots before and after running code, then compare them to find the lines allocating the most memory. The memory_profiler package with its @profile decorator also shows per-line memory consumption when run via python -m memory_profiler.

Use multiprocessing for CPU-bound tasks like heavy calculations, since it bypasses the GIL by running across multiple processes. Use asyncio with libraries like aiohttp for I/O-bound work such as concurrent HTTP requests, where tasks spend time waiting rather than computing.

Common causes include optimizing without profiling first, using lists where dict lookups are needed, excessive function call overhead in hot loops, and unnecessary data copies. Always measure with timeit or cProfile to confirm the actual bottleneck before changing code.