classification
Title: Incorrect iteration of Manager.dict() method of the multiprocessing module.
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, davin, deltaclock, miss-islington, pitrou, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2018-09-07 23:43 by deltaclock, last changed 2018-09-22 18:35 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
test.py deltaclock, 2018-09-07 23:43 Proof of concept code.
Pull Requests
URL Status Linked Edit
PR 9113 merged serhiy.storchaka, 2018-09-08 10:01
PR 9349 merged miss-islington, 2018-09-17 11:24
PR 9350 merged miss-islington, 2018-09-17 11:24
PR 9500 merged serhiy.storchaka, 2018-09-22 18:16
Messages (6)
msg324813 - (view) Author: (deltaclock) Date: 2018-09-07 23:43
The manager class of the multiprocessing module doesnt implement __next__ or __iter__ on its dictionary method and relies on using __getitem__.

This results in an unexpected functionality of an object that is supposed to act like a dict, and instead of looping over the dictionary keys it loops over its values.

The user needs to call dict() on this object in order to loop over it correctly, which is not that obvious.

The problem originates from there https://github.com/python/cpython/blob/master/Lib/multiprocessing/managers.py#L1136

Poc link: https://hastebin.com/codoqihowi.py
msg324820 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-09-08 08:08
DictProxy needs to provide the __iter__() method. And perhaps methods keys(), values() and items() need to be reworked in Python 3. has_key() is unneeded in Python 3.

It is worth to test also other special and new in Python 3 methods.
msg325522 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-09-17 11:24
New changeset e0e5065daef36dafe10a46eaa8b7800274d73062 by Serhiy Storchaka in branch 'master':
bpo-34610: Fixed iterator of multiprocessing.managers.DictProxy. (GH-9113)
https://github.com/python/cpython/commit/e0e5065daef36dafe10a46eaa8b7800274d73062
msg325528 - (view) Author: miss-islington (miss-islington) Date: 2018-09-17 12:11
New changeset 1d3078849381335008379e3de96d5a81d2131113 by Miss Islington (bot) in branch '3.7':
bpo-34610: Fixed iterator of multiprocessing.managers.DictProxy. (GH-9113)
https://github.com/python/cpython/commit/1d3078849381335008379e3de96d5a81d2131113
msg325529 - (view) Author: miss-islington (miss-islington) Date: 2018-09-17 12:11
New changeset f35e4d5851555d66f1e7a36b2a37f0d82b3d1706 by Miss Islington (bot) in branch '3.6':
bpo-34610: Fixed iterator of multiprocessing.managers.DictProxy. (GH-9113)
https://github.com/python/cpython/commit/f35e4d5851555d66f1e7a36b2a37f0d82b3d1706
msg326107 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-09-22 18:34
New changeset 69d0bc1430d2e9cddf0b39054ddcb86dbbe7927e by Serhiy Storchaka in branch '2.7':
[2.7] bpo-34610: Fixed iterator of multiprocessing.managers.DictProxy. (GH-9113). (GH-9500)
https://github.com/python/cpython/commit/69d0bc1430d2e9cddf0b39054ddcb86dbbe7927e
History
Date User Action Args
2018-09-22 18:35:26serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-09-22 18:34:26serhiy.storchakasetmessages: + msg326107
2018-09-22 18:16:29serhiy.storchakasetpull_requests: + pull_request8907
2018-09-17 12:11:07miss-islingtonsetmessages: + msg325529
2018-09-17 12:11:02miss-islingtonsetnosy: + miss-islington
messages: + msg325528
2018-09-17 11:24:38miss-islingtonsetpull_requests: + pull_request8773
2018-09-17 11:24:27miss-islingtonsetpull_requests: + pull_request8772
2018-09-17 11:24:06serhiy.storchakasetmessages: + msg325522
2018-09-08 10:01:33serhiy.storchakasetnosy: + pitrou, davin
2018-09-08 10:01:11serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request8567
2018-09-08 08:08:20serhiy.storchakasetnosy: + benjamin.peterson, serhiy.storchaka

messages: + msg324820
versions: + Python 2.7, Python 3.7, Python 3.8
2018-09-07 23:43:37deltaclockcreate