Profile Script with profiling.sampling in Python 3.15

A slow script rarely shows what's causing the delay. Guessing by reading the code may work for small programs, but the problem often hides in a line that looks harmless. Since Python 3.15, the profiling package provides the profiling.sampling module, which is a sampling profiler. It takes periodic snapshots of the call stack of a running program, then reports where the program spent its time.

The following script collects the unique names from a list of records:

rows = [(f'user{i % 500}', i) for i in range(200000)]


def unique_names(records):
    names = []
    for name, _ in records:
        if name not in names:
            names.append(name)
    return names


print(len(unique_names(rows)))

The run command starts the script and profiles it:

python -m profiling.sampling run main.py

500
Captured 489 samples in 0.49 seconds
Sample rate: 1000.00 samples/sec
Error rate: 3.48
Profile Stats:
       nsamples   sample%  tottime (ms)    cumul%  cumtime (ms)  filename:lineno(function)
        438/438      92.8       438.000      92.8       438.000  main.py:7(unique_names)
          33/33       7.0        33.000       7.0        33.000  main.py:1(<module>)
            1/1       0.2         1.000       0.2         1.000  main.py:6(unique_names)
          0/472       0.0         0.000     100.0       472.000  <frozen runpy>:87(_run_code)
          0/472       0.0         0.000     100.0       472.000  <frozen runpy>:201(_run_module_as_main)
          0/439       0.0         0.000      93.0       439.000  main.py:12(<module>)

The script runs normally and prints 500, then the profiler prints its report.

The number after the file name is the line that was executing when the sample was taken, not the line where the function is defined. Line 7 is if name not in names, and it accounts for 92.8% of the time. Line 6 is the for statement, and line 1 is the list comprehension that builds rows. The two <frozen runpy> rows show how Python started the script.

The names variable is a list, and in on a list compares the elements one by one. The list grows to 500 entries and the test runs once per row, so the loop performs up to 200,000 × 500 comparisons.

A set makes the check fast, and the list is kept to preserve the order of the names:

rows = [(f'user{i % 500}', i) for i in range(200000)]


def unique_names(records):
    names = []
    seen = set()
    for name, _ in records:
        if name not in seen:
            seen.add(name)
            names.append(name)
    return names


print(len(unique_names(rows)))

Run the command again:

python -m profiling.sampling run main.py

500
Captured 54 samples in 0.05 seconds
Sample rate: 1000.00 samples/sec
Error rate: 29.63
Profile Stats:
       nsamples   sample%  tottime (ms)    cumul%  cumtime (ms)  filename:lineno(function)
          31/31      81.6        31.000      81.6        31.000  main.py:1(<module>)
            5/5      13.2         5.000      13.2         5.000  main.py:8(unique_names)
            2/2       5.3         2.000       5.3         2.000  main.py:7(unique_names)
           0/38       0.0         0.000     100.0        38.000  <frozen runpy>:87(_run_code)
           0/38       0.0         0.000     100.0        38.000  <frozen runpy>:201(_run_module_as_main)
            0/7       0.0         0.000      18.4         7.000  main.py:14(<module>)

The script now finishes in 0.05 seconds instead of 0.49. The membership test on line 8 now costs 13.2%, and the largest entry is line 1, which builds the test data.

The --flamegraph option writes the same data as an interactive chart instead of a table:

python -m profiling.sampling run --flamegraph -o flame.html main.py

500
Captured 60 samples in 0.06 seconds
Sample rate: 1000.00 samples/sec
Error rate: 30.00
Flamegraph data: 1 root function, 42 total samples, 28 unique strings
Flamegraph saved to: flame.html

The result is a single HTML file that opens in a browser with no server and no other tools.

Profiling a script this way needs no special permissions, because the profiler starts the script itself.