classification
Title: WithProcessesTestSharedMemory.test_shared_memory_basics fails on Windows
Type: behavior Stage: resolved
Components: Library (Lib), Tests, Windows Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: davin, eryksun, miss-islington, pablogsal, paul.moore, pitrou, serhiy.storchaka, sobolevn, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2021-08-30 18:21 by sobolevn, last changed 2021-09-07 14:34 by pablogsal. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 28182 merged sobolevn, 2021-09-06 16:01
PR 28185 merged miss-islington, 2021-09-06 16:55
Messages (13)
msg400646 - (view) Author: Nikita Sobolev (sobolevn) * Date: 2021-08-30 18:21
While working on https://github.com/python/cpython/pull/28060 we've noticed that `test.test_multiprocessing_spawn.WithProcessesTestSharedMemory.test_shared_memory_basics` fails on Windows:

```
======================================================================
FAIL: test_shared_memory_basics (test.test_multiprocessing_spawn.WithProcessesTestSharedMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\cpython\cpython\lib\test\_test_multiprocessing.py", line 3794, in test_shared_memory_basics
    self.assertEqual(sms.size, sms2.size)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 512 != 4096
```

For now it is ignored.

Related issue: https://bugs.python.org/issue45042
msg400647 - (view) Author: Nikita Sobolev (sobolevn) * Date: 2021-08-30 18:21
I would like to work on this issue.
msg400659 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-08-30 19:00
It may be a bug in the constructor of SharedMemory. It ignores the size argument on Windows.
msg400689 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-08-31 04:42
> It may be a bug in the constructor of SharedMemory. It ignores 
> the size argument on Windows.

The `size` argument is always ignored when `create` is false, on all platforms, not that I understand the reason for it. The difference compared to POSIX is that the size of a mapping that's backed by the paging file is rounded up to a multiple of the page size in Windows (e.g. 4 KiB).
msg401146 - (view) Author: Nikita Sobolev (sobolevn) * Date: 2021-09-06 15:49
Ok, that's what I was able to find.

Original BPO: https://bugs.python.org/issue35813
Original PRs (sources + tests):
- https://github.com/python/cpython/pull/11664 it was a little bit "rushed"
- https://github.com/python/cpython/pull/11816 follow up

Related: https://bugs.python.org/issue38169

1. The test fails because `sms.size != sms2.size`
2. The `sms2` object is a pickle-unpickle copy of `sms`
3. The `__reduce__` method defined in `SharedMemory` returns `(name=self.name, create=False, size=self.size)` tuple
4. So, when `sms2` is constructed, `size` is ignored, because `create` is `False`

Moreover, even when `size` is passed with `create=True` the docs say that the final amount of allocated memory is platform specific (no details on the exact platforms): it can be larger or equal. Link: https://github.com/python/cpython/blame/main/Doc/library/multiprocessing.shared_memory.rst#L61 

So, this statement makes `self.assertEqual(sms.size, sms2.size)` potentially flaky.

After reading all this my suggestion is:
1. Remove `self.assertEqual(sms.size, sms2.size)` line, since this behavior is not guaranteed to work
2. Add docs about how `pickle` works with `SharedMemory` / `ShareableList`. Right now it is not clear. This should be just an explanation of the current status-quo
3. Add more tests on unpickled `Shared*` objects. Including overflowing set memory limits on unpickled objects and other implemetation details

I will start with `1.` right now, later I can work on `3.`.
But, I feel like someone more knowledgeable should take `2.` (or at least guide me, since `SharedMemory` is not something I use everyday).
msg401157 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-09-06 16:55
New changeset 19871fce3b74fc3f37e334a999e00d0ef65a8f1e by Nikita Sobolev in branch 'main':
bpo-45052: Unskips a failing `test_shared_memory_basics` test (GH-28182)
https://github.com/python/cpython/commit/19871fce3b74fc3f37e334a999e00d0ef65a8f1e
msg401158 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-09-06 16:55
The test fix looks good to me. That resolves this issue, yes? The other work is going on elsewhere?
msg401159 - (view) Author: Nikita Sobolev (sobolevn) * Date: 2021-09-06 17:01
I think that adding extra tests and docs are two separate new tasks. I will
open them today.

Thanks a lot, everyone!

пн, 6 сент. 2021 г. в 19:56, Steve Dower <report@bugs.python.org>:

>
> Steve Dower <steve.dower@python.org> added the comment:
>
> The test fix looks good to me. That resolves this issue, yes? The other
> work is going on elsewhere?
>
> ----------
> nosy:  -miss-islington
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue45052>
> _______________________________________
>
msg401164 - (view) Author: miss-islington (miss-islington) Date: 2021-09-06 17:16
New changeset 6b5aea2dc1bf7e117d40d6c9035d5c13124fd968 by Miss Islington (bot) in branch '3.10':
bpo-45052: Unskips a failing `test_shared_memory_basics` test (GH-28182)
https://github.com/python/cpython/commit/6b5aea2dc1bf7e117d40d6c9035d5c13124fd968
msg401220 - (view) Author: Nikita Sobolev (sobolevn) * Date: 2021-09-07 10:18
Closing and moving to https://bugs.python.org/issue45125
msg401263 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-07 13:19
This seem to have caused several errors that have manifested on the release of 3.10.0rc2:

test test_multiprocessing_fork failed -- Traceback (most recent call last):
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1239, in _dot_lookup
    return getattr(thing, comp)
AttributeError: module 'multiprocessing' has no attribute 'shared_memory'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/test/_test_multiprocessing.py", line 3818, in test_shared_memory_basics
    with unittest.mock.patch(
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1422, in __enter__
    self.target = self.getter()
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1609, in <lambda>
    getter = lambda: _importer(target)
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1252, in _importer
    thing = _dot_lookup(thing, comp, import_path)
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1242, in _dot_lookup
    return getattr(thing, comp)
AttributeError: module 'multiprocessing' has no attribute 'shared_memory'

0:09:11 load avg: 0.71 [231/427/1] test_multiprocessing_forkserver -- test_multiprocessing_fork failed (1 error) in 1 min 11 sec
test test_multiprocessing_forkserver failed -- Traceback (most recent call last):
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1239, in _dot_lookup
    return getattr(thing, comp)
AttributeError: module 'multiprocessing' has no attribute 'shared_memory'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/test/_test_multiprocessing.py", line 3818, in test_shared_memory_basics
    with unittest.mock.patch(
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1422, in __enter__
    self.target = self.getter()
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1609, in <lambda>
    getter = lambda: _importer(target)
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1252, in _importer
    thing = _dot_lookup(thing, comp, import_path)
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1242, in _dot_lookup
    return getattr(thing, comp)
AttributeError: module 'multiprocessing' has no attribute 'shared_memory'

0:11:11 load avg: 0.93 [232/427/2] test_multiprocessing_main_handling -- test_multiprocessing_forkserver failed (1 error) in 2 min
0:11:18 load avg: 1.09 [233/427/2] test_multiprocessing_spawn
test test_multiprocessing_spawn failed -- Traceback (most recent call last):
  File "/tmp/tmpu30qfjpr/installation/lib/python3.10/unittest/mock.py", line 1239, in _dot_lookup
    return getattr(thing, comp)
AttributeError: module 'multiprocessing' has no attribute 'shared_memory'
msg401275 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-07 13:54
I'm planning to revert PR 28185 because this is blocking the release
msg401288 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-07 14:34
Opened https://bugs.python.org/issue45128
History
Date User Action Args
2021-09-07 14:34:08pablogsalsetmessages: + msg401288
2021-09-07 13:54:12pablogsalsetmessages: + msg401275
2021-09-07 13:19:04pablogsalsetnosy: + pablogsal
messages: + msg401263
2021-09-07 10:18:13sobolevnsetstatus: open -> closed
resolution: fixed
messages: + msg401220

stage: patch review -> resolved
2021-09-06 17:16:51miss-islingtonsetnosy: + miss-islington
messages: + msg401164
2021-09-06 17:01:44sobolevnsetmessages: + msg401159
2021-09-06 16:55:56steve.dowersetnosy: - miss-islington
messages: + msg401158
2021-09-06 16:55:53miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request26613
2021-09-06 16:55:41steve.dowersetmessages: + msg401157
2021-09-06 16:01:18sobolevnsetkeywords: + patch
stage: patch review
pull_requests: + pull_request26611
2021-09-06 15:49:50sobolevnsetmessages: + msg401146
2021-08-31 04:42:06eryksunsetnosy: + eryksun
messages: + msg400689
2021-08-30 19:00:55serhiy.storchakasetnosy: + zach.ware, pitrou, paul.moore, tim.golden, serhiy.storchaka, davin, steve.dower
messages: + msg400659
components: + Library (Lib), Windows
2021-08-30 18:21:45sobolevnsetmessages: + msg400647
2021-08-30 18:21:28sobolevncreate