Skip to content

gh-129005: Remove copy in _pyio.FileIO.readall() - #129496

Closed
cmaloney wants to merge 1 commit into
python:mainfrom
cmaloney:pyio_readall_match_mem
Closed

gh-129005: Remove copy in _pyio.FileIO.readall()#129496
cmaloney wants to merge 1 commit into
python:mainfrom
cmaloney:pyio_readall_match_mem

Conversation

@cmaloney

@cmaloney cmaloney commented Jan 31, 2025

Copy link
Copy Markdown
Contributor

This aligns the memory usage between _pyio and _io. Both now use the same amount of memory when reading a file.

On my linux dev box, drops ./python -m test -M8g -uall test_largefile -m test_large_read -v from ~3.3 to ~2.4 seconds.

./python -m test -M8g -uall test_largefile -m test_large_read -v
Checked 112 modules (34 built-in, 77 shared, 1 n/a on linux-x86_64, 0 disabled, 0 missing, 0 failed on import)
== CPython 3.14.0a4+ (heads/pyio_readall_match_mem-dirty:9d23eb4375b, Jan 30 2025, 20:33:11) [Clang 19.1.7 ]
== Linux-6.12.10-arch1-1-x86_64-with-glibc2.40 little-endian
== Python build: debug
== cwd: <workdir>/python/build/build/test_python_worker_486953æ
== CPU count: 32
== encodings: locale=UTF-8 FS=utf-8
== resources: all

Using random seed: 4286629456
0:00:00 load avg: 1.00 Run 1 test sequentially in a single process
0:00:00 load avg: 1.00 [1/1] test_largefile
test_large_read (test.test_largefile.CLargeFileTest.test_large_read) ... 
 ... expected peak memory use: 2.3G
 ... process data size: 2.3G
ok
test_large_read (test.test_largefile.PyLargeFileTest.test_large_read) ... 
 ... expected peak memory use: 2.3G
 ... process data size: 2.3G
 ... process data size: 2.3G
ok

----------------------------------------------------------------------
Ran 2 tests in 2.353s

OK

== Tests result: SUCCESS ==

1 test OK.

Total duration: 2.4 sec
Total tests: run=2 (filtered)
Total test files: run=1/1 (filtered)
Result: SUCCESS
This aligns the memory usage between _pyio and _io. Both now use the
same amount of memory when reading a file.
@cmaloney

cmaloney commented Feb 2, 2025

Copy link
Copy Markdown
Contributor Author

I think this is the wrong way to approach this. Going to work on trying to make a zero-copy conversion from bytearray -> bytes instead, I think solves more generally for code, and creates less slight API differences that can break things.

@cmaloney cmaloney closed this Feb 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant