New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test_multiprocessing causes test_ctypes to fail #47375
Comments
test_ctypes, when run after testmultiprocessing, fails: ... Traceback (most recent call last):
File "c:\svn\trunk\lib\ctypes\test\test_pickling.py", line 29, in
test_simple
dst = self.loads(self.dumps(src))
File "c:\svn\trunk\lib\ctypes\test\test_pickling.py", line 19, in dumps
return pickle.dumps(item)
File "c:\svn\trunk\lib\pickle.py", line 1366, in dumps
Pickler(file, protocol).dump(obj)
File "c:\svn\trunk\lib\pickle.py", line 224, in dump
self.save(obj)
File "c:\svn\trunk\lib\pickle.py", line 301, in save
rv = reduce(obj)
File "c:\svn\trunk\lib\multiprocessing\sharedctypes.py", line 121, in
reduce_ctype
assert_spawning(obj)
File "c:\svn\trunk\lib\multiprocessing\forking.py", line 25, in
assert_spawning
' through inheritance' % type(self).__name__
RuntimeError: c_long objects should only be shared between processes
through inheritance ====================================================================== Traceback (most recent call last):
File "c:\svn\trunk\lib\ctypes\test\test_pickling.py", line 29, in
test_simple
dst = self.loads(self.dumps(src))
File "c:\svn\trunk\lib\ctypes\test\test_pickling.py", line 71, in dumps
return pickle.dumps(item, 1)
File "c:\svn\trunk\lib\pickle.py", line 1366, in dumps
Pickler(file, protocol).dump(obj)
File "c:\svn\trunk\lib\pickle.py", line 224, in dump
self.save(obj)
File "c:\svn\trunk\lib\pickle.py", line 301, in save
rv = reduce(obj)
File "c:\svn\trunk\lib\multiprocessing\sharedctypes.py", line 121, in
reduce_ctype
assert_spawning(obj)
File "c:\svn\trunk\lib\multiprocessing\forking.py", line 25, in
assert_spawning
' through inheritance' % type(self).__name__
RuntimeError: c_long objects should only be shared between processes
through inheritance ====================================================================== Traceback (most recent call last):
File "c:\svn\trunk\lib\ctypes\test\test_pickling.py", line 29, in
test_simple
dst = self.loads(self.dumps(src))
File "c:\svn\trunk\lib\ctypes\test\test_pickling.py", line 75, in dumps
return pickle.dumps(item, 2)
File "c:\svn\trunk\lib\pickle.py", line 1366, in dumps
Pickler(file, protocol).dump(obj)
File "c:\svn\trunk\lib\pickle.py", line 224, in dump
self.save(obj)
File "c:\svn\trunk\lib\pickle.py", line 301, in save
rv = reduce(obj)
File "c:\svn\trunk\lib\multiprocessing\sharedctypes.py", line 121, in
reduce_ctype
assert_spawning(obj)
File "c:\svn\trunk\lib\multiprocessing\forking.py", line 25, in
assert_spawning
' through inheritance' % type(self).__name__
RuntimeError: c_long objects should only be shared between processes
through inheritance |
Adam, is this due to the c-code tainting you pointed out earlier? |
IMO this problem occurs because multiprocessing registers pickling |
Jesse, can you be more specific? Thomas, do you have a specific command to reproduce this? It runs fine |
Sorry Adam, I was referencing: http://bugs.python.org/issue3102 Also, it is possible issue: http://bugs.python.org/issue3102 is related? |
Sorry, the first link should have been: |
Adam Olsen schrieb:
It seems the failure only occurs on Windows. On linux it does work for me too. See the traceback that I posted, and this: http://www.python.org/dev/buildbot/trunk/x86%20XP-3%20trunk/builds/1606/step-test/0 |
I see no common symbols between bpo-3102 and bpo-3092, so unless I missed I second the notion that multiprocessing's use of pickle is the I do see some win32-specific behaviour, which may be broken. Thomas, if sys.platform == 'win32' and type_ not in copy_reg.dispatch_table:
copy_reg.pickle(type_, reduce_ctype) |
This fixes the failure in test_ctypes, but test_multiprocessing no longer works: c:\svn\trunk\PCbuild>.\\python_d -E -tt ../lib/test/regrtest.py test_multiprocessing test_ctypes
test_multiprocessing
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "c:\svn\trunk\lib\multiprocessing\forking.py", line 297, in main
self = load(from_parent)
EOFError
[48274 refs]
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "c:\svn\trunk\lib\multiprocessing\forking.py", line 297, in main
self = load(from_parent)
EOFError
[48763 refs]
test test_multiprocessing failed -- errors occurred; run in verbose mode for details
test_ctypes
1 test OK.
1 test failed:
test_multiprocessing
[118894 refs] c:\svn\trunk\PCbuild> |
This patch to sharedctypes should fix the problem by adding a |
But why this is win32 specific? Is it because windows cannot fork(), so data has to be copied through |
Yes, on Windows pickling is needed to pass data to a child process. In |
roudkerk schrieb:
I can confirm that the patch fixes the problem on Windows (running test_ctypes However, the patch did not apply cleanly to current SVN trunk - I had to manually |
Thomas' patch applied and runs clean for me - does anyone have a problem |
roudkerk wrote:
I am not sure to understand. Can you elaborate? |
Removing the "if win32" bits will not make shared ctypes objects I do not want to encourage people to try to transfer objects which The simplest way to avoid such problems is to only share such |
Why not use a custom Pickler in this case? |
Here is a patch that implements a custom pickler for |
Jesse: ping? |
Sorry - I've been sick and overly busy this week, the mp issues are on On Jun 26, 2008, at 7:17 PM, Amaury Forgeot d'Arc <report@bugs.python.org
|
Tests are back on as of r64663 on trunk. |
I closed the wrong bug |
Amaury's patch is applied in r65016 to trunk, all tests pass This needs to be merged forward, and then Lib/multiprocessing/managers.py in py3k has to have: for view_type in view_types:
copy_reg.pickle(view_type, rebuild_as_list) changed to: for view_type in view_types:
ForkingPickler.register(view_type, rebuild_as_list) |
Here is a proposed patch for py3k generated from an svn merge of amaury's |
Working on the py3k patch now, bumping to rel. blocker |
Jesse, The only remaining point is the handling of dictionary views |
Just to confirm this issue was resolved enough to be lowered from a |
Is this still a problem or can this be closed, given r65883? |
As no one has been able to confirm that this is still an issue, I'm closing it as "out of date". The issue can be reopened if necessary. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: