gh-155525: Fix quadratic complexity in f-string tokenization

[gwosti]

Retain the start of each f-string and t-string replacement expression in the tokenizer buffer instead of allocating a temporary buffer and copying the remaining source for every replacement field. If the tokenizer buffer is resized, preserve the expression start as an offset and restore it afterward.

Microbenchmarks

• Optimized builds configured with --enable-optimizations --with-lto, Linux x86-64
• Pinned to one CPU; one warmup and seven measured runs, medians

Benchmark	workload	main	this PR	speedup
compile()	800 f-strings / 100 fields	1.314s	0.300s	4.38
ast.parse()	800 f-strings / 100 fields	1.474s	0.369s	4.00
geomean				4.18

Benchmark script

import ast
import gc
import statistics
import time

ROWS = (200, 400, 800)
FIELDS_PER_FSTRING = 100
RUNS = 7


def make_source(rows):
    fields = "".join(f"{{x{i}}}" for i in range(FIELDS_PER_FSTRING))
    return "".join(
        f"value_{i} = f'{fields}'\n"
        for i in range(rows)
    )


def compile_source(source):
    return compile(source, "<benchmark>", "exec")


def parse_source(source):
    return ast.parse(source, "<benchmark>", "exec")


def benchmark(func, source):
    func(source)  # Warmup
    samples = []

    for _ in range(RUNS):
        gc.collect()
        start = time.perf_counter()
        func(source)
        samples.append(time.perf_counter() - start)

    return statistics.median(samples), min(samples), max(samples)


for rows in ROWS:
    source = make_source(rows)

    for name, func in (
        ("compile()", compile_source),
        ("ast.parse()", parse_source),
    ):
        median, minimum, maximum = benchmark(func, source)
        print(
            f"{rows=:4} {name:11} "
            f"median={median:.6f}s "
            f"range=[{minimum:.6f}s, {maximum:.6f}s]"
        )

• Issue: Quadratic compile() / ast.parse() time for modules with many f-strings (PEP 701 tokenizer update_fstring_expr rescans the whole remaining buffer per replacement field) #155525

[python-cla-bot]

All commit authors signed the Contributor License Agreement.

[pablogsal]

Unless I am missing something, this also changes behaviour for a : after a !=: on main the ':' case only set last_expr_end when it was still -1, so for f'{a!=b=:>10}' the debug text was cut at the ! and t'{a!=b:>10}'.interpolations[0].expression was 'a'. Now we always take everything up to tok->start, which is the right thing, but can we add a test for both cases so we don't lose it?

[pablogsal]

Nit: this only checks that compile does not blow up, so a regression here would only show up as a slow test. Maybe we can exec it with the x* names defined and check a couple of the value_* results (same for the t-string one)?

[pablogsal]

Nit: can we add a comment here saying that this points into tok->buf and relies on _PyTok_ReaderUnderflow never resetting the buffer while INSIDE_FSTRING(tok)? That invariant is what makes this work and it lives a bit far from here.