Optimal Python Code — From Profiling Through Refactoring to Memory Efficiency

Before you optimize: measure, don’t guess

Optimizing Python code without profiling is shooting in the dark — intuition often points to the wrong places. Before you touch refactoring, run cProfile and see which functions actually dominate execution time.

# Run cProfile from command line:
python -m cProfile -o output.prof my_script.py

# Analyze results in Python:
import pstats
stats = pstats.Stats('output.prof')
stats.sort_stats('cumulative').print_stats(10)

cProfile is Python’s built-in function profiler that measures call statistics — execution time of each function and number of calls. It gives an overview, but won’t show which lines inside functions are slow.

When cProfile, when line_profiler?

Use cProfile as a first step — it identifies the 3-5 slowest functions. Then line_profiler analyzes those functions line-by-line, showing exactly where the bottleneck lies.

# Install line_profiler:
# pip install line_profiler

# Decorate function for line-by-line profiling:
@profile
def process_data(items):
    result = []
    for item in items:
        if item % 2 == 0:
            result.append(item * 2)
    return result

# Run with kernprof:
# kernprof -l -v my_script.py

The workflow is simple: cProfile gives the big picture, line_profiler shows details inside slow functions, then you optimize only what actually hurts.

Generators and itertools — memory efficiency in practice

A list of 10 million records uses hundreds of megabytes; a generator producing the same data uses kilobytes. The difference is that generators don’t build the entire structure in memory — they produce values on demand.

# BAD version — builds entire list in memory:
def process_with_list(n):
    result = []
    for i in range(n):
        result.append(i * 2)
    return result

# GOOD version — generator using constant memory:
def process_with_generator(n):
    for i in range(n):
        yield i * 2

# Usage:
for value in process_with_generator(10_000_000):
    process(value)  # Stream processing

Generator expressions are list comprehensions syntax with parentheses instead of brackets. They’re lazy — they don’t build a list in memory. Use them when you process data once, top to bottom, and memory matters.

# List comprehension — builds a list:
squares_list = [x**2 for x in range(1000000)]

# Generator expression — uses O(1) memory:
squares_gen = (x**2 for x in range(1000000))

# itertools — all functions return iterators, never lists:
from itertools import islice, chain, filterfalse

# Infinite iterator, safe because islice cuts it:
for x in islice(count(), 100):
    process(x)

# Combining iterators without building a list:
all_data = chain(iterator1, iterator2, iterator3)

The itertools module is a set of high-performance, memory-efficient building blocks for constructing iterators. All functions return iterators, so you can safely compose them even with infinite inputs.

Common performance pitfalls

String concatenation in a loop

The += operator on strings in a loop creates a new string object on each iteration — O(n2) cost. Instead, use "".join(), which allocates memory once.

# BAD — O(n2) allocations:
def concat_bad(items):
    result = ""
    for item in items:
        result += str(item)
    return result

# GOOD — O(n) allocations:
def concat_good(items):
    return "".join(str(item) for item in items)

List comprehension vs map/filter

In Python 3, map() and filter() return iterators, but list comprehension is more readable and often faster. Use map/filter only with existing functions, not lambdas.

# List comprehension — readable and fast:
even_doubled = [x*2 for x in data if x % 2 == 0]

# Avoid — lambda in map/filter is less readable:
even_doubled = list(map(lambda x: x*2, filter(lambda x: x % 2 == 0, data)))

# Acceptable — when function already exists:
even_doubled = list(map(double, filter(is_even, data)))

Tools beyond profiling: code quality

Optimal code isn’t just fast code — it’s also readable, maintainable, and free of type errors. PEP 8 is the official Python style guide. Black auto-formats code without debate. Ruff is a fast linter replacing slower tools. mypy statically checks types.

# Type hints + mypy:
from typing import List, Iterator

def process_items(items: List[int]) -> Iterator[int]:
    for item in items:
        if item % 2 == 0:
            yield item * 2

# Run mypy:
# mypy my_module.py

When (and when not) to optimize

The rule is simple: first make the code work, then make it readable, then make it fast — but only where profiling shows a problem. Optimization without data is a waste of time.

If a function is called 10 times and takes 0.1ms, there’s no point spending 3 hours speeding it up to 0.05ms. If another function is called a million times and takes 50ms, that’s where the real gain lies.

Alternatives to cProfile and line_profiler

For production profiling, use py-spy — it’s a sampling profiler with low overhead that works on running processes without restart. memory_profiler tracks memory usage line-by-line, similar to how line_profiler tracks time.

# py-spy — profile process in production:
py-spy top --pid 12345

# memory_profiler — memory line-by-line:
# pip install memory_profiler
# Run: python -m memory_profiler my_script.py

Summary — optimization workflow

• Step 1: Run cProfile — find the 3-5 slowest functions.
• Step 2: Use line_profiler on those functions — find slow lines.
• Step 3: Think about structure — can a list be a generator?
• Step 4: Check memory usage with memory_profiler.
• Step 5: Run profiling again — confirm improvement.

Optimal Python code is code that was first measured, then touched. The tools are there — cProfile, line_profiler, memory_profiler, py-spy — use them in that order, and you’ll stop guessing and start optimizing where it really matters.

Where to go next

If you want to understand what’s happening under Python’s hood, start with the article on malloc internals — it shows why free() doesn’t return memory to the system and how that affects memory profiling. For debugging bottlenecks, strace will help — when profiling shows a syscall as a bottleneck, strace will show exactly what’s happening. And if you’re interested in I/O optimization, epoll vs io_uring will show when the event loop isn’t enough.