Skip to content

gh-129005: Remove copies from _pyio using take_bytes - #141539

Merged
vstinner merged 2 commits into
python:mainfrom
cmaloney:gh129005_take_bytes
Nov 18, 2025
Merged

gh-129005: Remove copies from _pyio using take_bytes#141539
vstinner merged 2 commits into
python:mainfrom
cmaloney:gh129005_take_bytes

Conversation

@cmaloney

@cmaloney cmaloney commented Nov 13, 2025

Copy link
Copy Markdown
Contributor

Memory usage now matches that of _io for large files.

$ ./python -Wd -I -m test test_io.test_largefile -uall,largefile,walltime -M32G -v
== CPython 3.15.0a1+ (heads/gh129005_take_bytes-dirty:d8b88498ebf, Nov 13 2025, 15:43:51) [Clang 21.1.5 ]
== Linux-6.17.7-arch1-1-x86_64-with-glibc2.42 little-endian
== Python build: debug
== cwd: <build_dir>/build/test_python_worker_116910æ
== CPU count: 32
== encodings: locale=UTF-8 FS=utf-8
== resources: all

Using random seed: 365280675
0:00:00 load avg: 9.60 Run 1 test sequentially in a single process
0:00:00 load avg: 9.60 [1/1] test_io.test_largefile
test_large_read (test.test_io.test_largefile.CLargeFileTest.test_large_read) ... 
 ... expected peak memory use: 2.3G
 ... process data size: 2.3G
ok
test_lseek (test.test_io.test_largefile.CLargeFileTest.test_lseek) ... ok
test_osstat (test.test_io.test_largefile.CLargeFileTest.test_osstat) ... ok
test_seek_read (test.test_io.test_largefile.CLargeFileTest.test_seek_read) ... ok
test_seek_readall (test.test_io.test_largefile.CLargeFileTest.test_seek_readall) ... 
 ... expected peak memory use: 2.3G
 ... process data size: 2.3G
 ... process data size: 2.3G
ok
test_seekable (test.test_io.test_largefile.CLargeFileTest.test_seekable) ... ok
test_truncate (test.test_io.test_largefile.CLargeFileTest.test_truncate) ... ok
test_large_read (test.test_io.test_largefile.PyLargeFileTest.test_large_read) ... 
 ... expected peak memory use: 2.3G
 ... process data size: 2.3G
ok
test_lseek (test.test_io.test_largefile.PyLargeFileTest.test_lseek) ... ok
test_osstat (test.test_io.test_largefile.PyLargeFileTest.test_osstat) ... ok
test_seek_read (test.test_io.test_largefile.PyLargeFileTest.test_seek_read) ... ok
test_seek_readall (test.test_io.test_largefile.PyLargeFileTest.test_seek_readall) ... 
 ... expected peak memory use: 2.3G
 ... process data size: 2.3G
 ... process data size: 2.3G
ok
test_seekable (test.test_io.test_largefile.PyLargeFileTest.test_seekable) ... ok
test_truncate (test.test_io.test_largefile.PyLargeFileTest.test_truncate) ... ok
test_it (test.test_io.test_largefile.TestCopyfile.test_it) ... ok
test_it (test.test_io.test_largefile.TestSocketSendfile.test_it) ... ok

----------------------------------------------------------------------
Ran 16 tests in 6.950s

OK
0:00:06 load avg: 9.72 [1/1] test_io.test_largefile passed

== Tests result: SUCCESS ==

1 test OK.

Total duration: 7.0 sec
Total tests: run=16
Total test files: run=1/1
Result: SUCCESS

For reference; this test goes from 8.5sec overall duration on my AMD 64 bit Arch Linux box to 7 seconds.

Memory usage now matches that of _io for large files
@cmaloney

Copy link
Copy Markdown
Contributor Author

Optimization enabled by gh-139871 (Trying to keep that one issue referencing all the things which I use it for; planning to expand the What's New with a single "here's all the modules that got faster" list once optimize)

@cmaloney cmaloney changed the title gh-129005: Remove copy from _pyio using take_bytes Nov 14, 2025
Comment thread Lib/_pyio.py
@@ -620,15 +620,15 @@ def read(self, size=-1):
if n < 0 or n > len(b):
raise ValueError(f"readinto returned {n} outside buffer size {len(b)}")
del b[n:]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a difference between b.resize(n) and del b[n:]?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.resize() could grow while the del b[n:] never will. There's some code which does a = bytearray(b'12'); del a[3:] (intentionally del a slice past the end) as part of its buffer management. Generally I prefer resize because there's a lot less code involved than the slice code (and the slice code eventually calls resize anyways)

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vstinner
vstinner merged commit 58f3fe0 into python:main Nov 18, 2025
48 checks passed
@cmaloney
cmaloney deleted the gh129005_take_bytes branch November 18, 2025 22:10
StanFromIreland pushed a commit to StanFromIreland/cpython that referenced this pull request Dec 6, 2025
…1539)

Memory usage now matches that of _io for large files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

2 participants