Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WithProcessesTestSharedMemory.test_shared_memory_basics fails on Windows #89215

Closed
sobolevn opened this issue Aug 30, 2021 · 13 comments
Closed

WithProcessesTestSharedMemory.test_shared_memory_basics fails on Windows #89215

sobolevn opened this issue Aug 30, 2021 · 13 comments
Labels
3.9 only security fixes 3.10 only security fixes 3.11 only security fixes OS-windows stdlib Python modules in the Lib dir tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error

Comments

@sobolevn
Copy link
Member

BPO 45052
Nosy @pfmoore, @pitrou, @tjguk, @zware, @serhiy-storchaka, @eryksun, @zooba, @applio, @pablogsal, @miss-islington, @sobolevn
PRs
  • bpo-45052: Unskips a failing test_shared_memory_basics test #28182
  • [3.10] bpo-45052: Unskips a failing test_shared_memory_basics test (GH-28182) #28185
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-09-07.10:18:13.014>
created_at = <Date 2021-08-30.18:21:28.942>
labels = ['type-bug', '3.9', '3.10', '3.11', 'library', 'tests', 'OS-windows']
title = 'WithProcessesTestSharedMemory.test_shared_memory_basics fails on Windows'
updated_at = <Date 2021-09-07.14:34:08.096>
user = 'https://github.com/sobolevn'

    bugs.python.org fields:

    activity = <Date 2021-09-07.14:34:08.096>
actor = 'pablogsal'
assignee = 'none'
closed = True
closed_date = <Date 2021-09-07.10:18:13.014>
closer = 'sobolevn'
components = ['Library (Lib)', 'Tests', 'Windows']
creation = <Date 2021-08-30.18:21:28.942>
creator = 'sobolevn'
dependencies = []
files = []
hgrepos = []
issue_num = 45052
keywords = ['patch']
message_count = 13.0
messages = ['400646', '400647', '400659', '400689', '401146', '401157', '401158', '401159', '401164', '401220', '401263', '401275', '401288']
nosy_count = 11.0
nosy_names = ['paul.moore', 'pitrou', 'tim.golden', 'zach.ware', 'serhiy.storchaka', 'eryksun', 'steve.dower', 'davin', 'pablogsal', 'miss-islington', 'sobolevn']
pr_nums = ['28182', '28185']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45052'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']
    @sobolevn
    Copy link
    Member Author

    While working on #28060 we've noticed that test.test_multiprocessing_spawn.WithProcessesTestSharedMemory.test_shared_memory_basics fails on Windows:

    ======================================================================
FAIL: test_shared_memory_basics (test.test_multiprocessing_spawn.WithProcessesTestSharedMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\cpython\cpython\lib\test\_test_multiprocessing.py", line 3794, in test_shared_memory_basics
    self.assertEqual(sms.size, sms2.size)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 512 != 4096

    For now it is ignored.

    Related issue: https://bugs.python.org/issue45042

    @sobolevn sobolevn added 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error labels Aug 30, 2021
    @sobolevn
    Copy link
    Member Author

    I would like to work on this issue.

    @serhiy-storchaka
    Copy link
    Member

    It may be a bug in the constructor of SharedMemory. It ignores the size argument on Windows.

    @serhiy-storchaka serhiy-storchaka added stdlib Python modules in the Lib dir OS-windows labels Aug 30, 2021
    @eryksun
    Copy link
    Contributor

    eryksun commented Aug 31, 2021

    It may be a bug in the constructor of SharedMemory. It ignores
    the size argument on Windows.

    The size argument is always ignored when create is false, on all platforms, not that I understand the reason for it. The difference compared to POSIX is that the size of a mapping that's backed by the paging file is rounded up to a multiple of the page size in Windows (e.g. 4 KiB).

    @sobolevn
    Copy link
    Member Author

    sobolevn commented Sep 6, 2021

    Ok, that's what I was able to find.

    Original BPO: https://bugs.python.org/issue35813
    Original PRs (sources + tests):

    Related: https://bugs.python.org/issue38169

    1. The test fails because sms.size != sms2.size
    2. The sms2 object is a pickle-unpickle copy of sms
    3. The __reduce__ method defined in SharedMemory returns (name=self.name, create=False, size=self.size) tuple
    4. So, when sms2 is constructed, size is ignored, because create is False

    Moreover, even when size is passed with create=True the docs say that the final amount of allocated memory is platform specific (no details on the exact platforms): it can be larger or equal. Link: https://github.com/python/cpython/blame/main/Doc/library/multiprocessing.shared_memory.rst#L61

    So, this statement makes self.assertEqual(sms.size, sms2.size) potentially flaky.

    After reading all this my suggestion is:

    1. Remove self.assertEqual(sms.size, sms2.size) line, since this behavior is not guaranteed to work
    2. Add docs about how pickle works with SharedMemory / ShareableList. Right now it is not clear. This should be just an explanation of the current status-quo
    3. Add more tests on unpickled Shared* objects. Including overflowing set memory limits on unpickled objects and other implemetation details

    I will start with 1. right now, later I can work on 3..
    But, I feel like someone more knowledgeable should take 2. (or at least guide me, since SharedMemory is not something I use everyday).

    @zooba
    Copy link
    Member

    zooba commented Sep 6, 2021

    New changeset 19871fc by Nikita Sobolev in branch 'main':
    bpo-45052: Unskips a failing test_shared_memory_basics test (GH-28182)
    19871fc

    @zooba
    Copy link
    Member

    zooba commented Sep 6, 2021

    The test fix looks good to me. That resolves this issue, yes? The other work is going on elsewhere?

    @sobolevn
    Copy link
    Member Author

    sobolevn commented Sep 6, 2021

    I think that adding extra tests and docs are two separate new tasks. I will
    open them today.

    Thanks a lot, everyone!

    пн, 6 сент. 2021 г. в 19:56, Steve Dower <report@bugs.python.org>:

    Steve Dower <steve.dower@python.org> added the comment:

    The test fix looks good to me. That resolves this issue, yes? The other
    work is going on elsewhere?

    ----------
    nosy: -miss-islington

    Python tracker <report@bugs.python.org>
    <https://bugs.python.org/issue45052\>

    @miss-islington
    Copy link
    Contributor

    New changeset 6b5aea2 by Miss Islington (bot) in branch '3.10':
    bpo-45052: Unskips a failing test_shared_memory_basics test (GH-28182)
    6b5aea2

    @sobolevn
    Copy link
    Member Author

    sobolevn commented Sep 7, 2021

    Closing and moving to https://bugs.python.org/issue45125

    @sobolevn sobolevn closed this as completed Sep 7, 2021
    @sobolevn sobolevn closed this as completed Sep 7, 2021
    @pablogsal
    Copy link
    Member

    This seem to have caused several errors that have manifested on the release of 3.10.0rc2:

    test test_multiprocessing_fork failed -- Traceback (most recent call last):
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1239, in _dot_lookup
    return getattr(thing, comp)
AttributeError: module 'multiprocessing' has no attribute 'shared_memory'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/test/_test_multiprocessing.py", line 3818, in test_shared_memory_basics
    with unittest.mock.patch(
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1422, in __enter__
    self.target = self.getter()
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1609, in <lambda>
    getter = lambda: _importer(target)
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1252, in _importer
    thing = _dot_lookup(thing, comp, import_path)
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1242, in _dot_lookup
    return getattr(thing, comp)
AttributeError: module 'multiprocessing' has no attribute 'shared_memory'

0:09:11 load avg: 0.71 [231/427/1] test_multiprocessing_forkserver -- test_multiprocessing_fork failed (1 error) in 1 min 11 sec
test test_multiprocessing_forkserver failed -- Traceback (most recent call last):
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1239, in _dot_lookup
    return getattr(thing, comp)
AttributeError: module 'multiprocessing' has no attribute 'shared_memory'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/test/_test_multiprocessing.py", line 3818, in test_shared_memory_basics
    with unittest.mock.patch(
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1422, in __enter__
    self.target = self.getter()
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1609, in <lambda>
    getter = lambda: _importer(target)
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1252, in _importer
    thing = _dot_lookup(thing, comp, import_path)
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1242, in _dot_lookup
    return getattr(thing, comp)
AttributeError: module 'multiprocessing' has no attribute 'shared_memory'

0:11:11 load avg: 0.93 [232/427/2] test_multiprocessing_main_handling -- test_multiprocessing_forkserver failed (1 error) in 2 min
0:11:18 load avg: 1.09 [233/427/2] test_multiprocessing_spawn
test test_multiprocessing_spawn failed -- Traceback (most recent call last):
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1239, in _dot_lookup
    return getattr(thing, comp)
AttributeError: module 'multiprocessing' has no attribute 'shared_memory'

    @pablogsal
    Copy link
    Member

    I'm planning to revert PR 28185 because this is blocking the release

    @pablogsal
    Copy link
    Member

    Opened https://bugs.python.org/issue45128

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.9 only security fixes 3.10 only security fixes 3.11 only security fixes OS-windows stdlib Python modules in the Lib dir tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    6 participants
    @eryksun @zooba @serhiy-storchaka @sobolevn @pablogsal @miss-islington