New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
3.0 file.read dreadfully slow #48783
Comments
C.l.p poster reported that 3.0 file.read is orders of magnitude slower "Here's a quick comparison between 2.5 and C:\>c:\Python30\python -m timeit -n 1 C:\>c:\Python25\python -m timeit -n 1 That's 3 orders of magnitude slower on python3.0!" I verified this informally on WinXP by opening and then reading
Doc/Pythonxy.chm (about 4 megs) -- an eye blink versus 3 seconds,
repeated. Even the open seemed slower but I did not time it.
>>> f=open('Doc/Python30.chm','rb')
>>> d=f.read() |
This needs definitely some testing! |
The small buffer size in Modules/_fileio.c is one reason for the slowness. $ dd if=/dev/zero of=zeros bs=1MB count=50
$ cat testread.py
open("zeros", "rb").read()
$ ./python -m cProfile testread.py
40 function calls (39 primitive calls) in 4.246 CPU seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function)
isinstance}
'_thread.lock' objects} $ vi Modules/_fileio.c
-#define DEFAULT_BUFFER_SIZE (8*1024)
+#define DEFAULT_BUFFER_SIZE (80*1024)
$ ./python -m cProfile testread.py
40 function calls (39 primitive calls) in 1.273 CPU seconds
Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) |
The fileio_buffer.patch implements the same progressive buffer as Python |
patch looks good to me. nitpick comments: use += instead of = and + in: newsize = newsize + newsize
and
newsize = newsize + BIGCHUNK. As for the XXX about overflow, so long as BUFSIZ is not defined to be an #if (BUFSIZ >= 2**30)
#error "unreasonable BUFSIZ defined"
#endif |
The preprocessor doesn't handle power. 2 << 24 (64MB) sounds sufficient |
fileio_buffer2.patch looks good other than minor touchups: Turn the XXX comment into: /* NOTE: overflow impossible due to limits on BUFSIZ * Also, 2 << 24 is 32MB yet your error message test says >= 64MB. I think fix those and commit. :) |
The updated patch has been committed to 3.0 and 3.1. I'm going to |
Since it is solved for 3.x and only needs to be bacported to 2.x (where |
This has been fixed as part of the big IO update in trunk. I assume |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: