New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Release GIL periodically in _pickle module #78309
Comments
Hi, the old and slow python implementation of pickle didn't block background Regards, |
um, something doesn't make sense about this. the python implementation of pickle never released the GIL (it can't, by definition -- it's written in python). The C implementation releasing the GIL wouldn't make sense, as the pickle api involves calls into python everywhere (for example, |
This is about releasing the GIL periodically to allow other threads to run, as Python already does in its main interpreter loop. |
A workaround is writing Python wrappers for IO: def Writer: def Reader: def mydump(obj, file, *args, **kwargs):
return pickle.dump(obj, Writer(file), *args, **kwargs)
def myload(file, *args, **kwargs):
return pickle.load(Reader(file), *args, **kwargs) |
Maybe an optional parameter with the desired interval would be good idea. So that the coder can decide if he wants/needs that feature and which interval he needs for his application. |
The right way to do this is not to pass a timeout parameter but to check for GIL interrupts as done in the main bytecode evaluation loop. |
Attaching proof-of-concept patch. |
Attaching demonstration script. |
(as the demo script shows, there is no detectable slowdown) |
The demo script shows around 8% slowdown to me for data = list(map(float, range(N))) |
Interesting, which kind of computer / system / compiler are you on? |
CPU = Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz
Ubuntu 18.04
Linux 4.15.0 x86_64
gcc 7.3.0 Performing the check in save() can have not insignificant overhead (especially after implementing the bpo-34141 optimization). It can be reduced if perform it when flush a frame (in protocol 4) or buffer to the file, or after writing significant amount of bytes into buffer. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: