python-performance-optimization

Use cProfile to profile Python code by wrapping execution in a Profile object or running python -m cProfile on your script. Sort results by cumulative time with pstats to see which functions consume the most execution time.

Use memory_profiler with the @profile decorator to see line-by-line memory consumption, or use tracemalloc to take snapshots before and after execution and compare allocations. Both reveal where large objects are created.

cProfile is a deterministic built-in profiler best for development and scripts, while py-spy is a sampling profiler that attaches to running production processes without code changes or restarts. Use py-spy for live systems and cProfile for local analysis.

Use multiprocessing for CPU-bound tasks like heavy computation, since it bypasses the GIL with separate processes. Use asyncio with libraries like aiohttp for I/O-bound work such as concurrent HTTP requests, where tasks wait on network or disk.

List membership tests are O(n) because Python scans every element. Convert the data to a dict or set for O(1) lookups, which can be orders of magnitude faster for repeated membership checks on large collections.

functools.lru_cache works on functions with hashable arguments and is most effective for deterministic, repeatedly-called functions like recursive algorithms. It does not help functions with side effects or unhashable arguments such as lists.