classification
Title: multiprocessing: test_shared_memory_across_processes() cannot be run twice in parallel
Type: Stage: resolved
Components: Tests Versions: Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: miss-islington, shihai1991, vstinner
Priority: normal Keywords: patch

Created on 2020-04-01 13:18 by vstinner, last changed 2020-05-04 15:25 by miss-islington. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 19892 merged shihai1991, 2020-05-04 02:58
PR 19903 merged miss-islington, 2020-05-04 15:06
Messages (6)
msg365462 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-01 13:18
Sometimes, I need to run multiprocessing tests multiple times in parallel to attempt to reproduce a race condition. Except that test_shared_memory_across_processes() fails in this case:

$ ./python -m test test_multiprocessing_spawn --fail-env-changed -v -j4 -F -m test_shared_memory_across_processes 

== CPython 3.9.0a5+ (heads/master:17b4733f2f, Apr 1 2020, 15:02:04) [GCC 9.2.1 20190827 (Red Hat 9.2.1-1)]
== Linux-5.5.9-200.fc31.x86_64-x86_64-with-glibc2.30 little-endian
== cwd: /home/vstinner/python/master/build/test_python_1133450
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 1.64 Run tests in parallel using 4 child processes
0:00:01 load avg: 1.64 [  1/1] test_multiprocessing_spawn failed
test_shared_memory_across_processes (test.test_multiprocessing_spawn.WithProcessesTestSharedMemory) ... ERROR

======================================================================
ERROR: test_shared_memory_across_processes (test.test_multiprocessing_spawn.WithProcessesTestSharedMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/master/Lib/test/_test_multiprocessing.py", line 3862, in test_shared_memory_across_processes
    sms = shared_memory.SharedMemory('test02_tsmap', True, size=512)
  File "/home/vstinner/python/master/Lib/multiprocessing/shared_memory.py", line 100, in __init__
    self._fd = _posixshmem.shm_open(
FileExistsError: [Errno 17] File exists: '/test02_tsmap'

----------------------------------------------------------------------

Ran 1 test in 0.247s

FAILED (errors=1)
test test_multiprocessing_spawn failed
Kill <TestWorkerProcess #2 running test=test_multiprocessing_spawn pid=1133458 time=1.2 sec> process group
Kill <TestWorkerProcess #3 running test=test_multiprocessing_spawn pid=1133457 time=1.2 sec> process group
Kill <TestWorkerProcess #4 running test=test_multiprocessing_spawn pid=1133459 time=1.2 sec> process group

== Tests result: FAILURE ==

1 test failed:
    test_multiprocessing_spawn

Total duration: 1.2 sec
Tests result: FAILURE
msg365463 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-01 13:21
Python 3.7 is not affected: it doesn't have test_shared_memory_across_processes().
msg368007 - (view) Author: hai shi (shihai1991) * (Python triager) Date: 2020-05-04 03:14
The easy way to fix this test case is that don't define the shared memory block's name. `test_shared_memory_basics` has tested against the shared memory block's name, so canceling the shared memory block'name has no side-effect.
msg368056 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-04 15:06
New changeset caa3ef284a2e5e5b9bdd6a9e619804122c842d80 by Hai Shi in branch 'master':
bpo-40135: Fix multiprocessing test_shared_memory_across_processes() (GH-19892)
https://github.com/python/cpython/commit/caa3ef284a2e5e5b9bdd6a9e619804122c842d80
msg368057 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-04 15:07
Thanks for the fix Hai Shi!
msg368059 - (view) Author: miss-islington (miss-islington) Date: 2020-05-04 15:25
New changeset 70fe95cdc9ac1b00d4f86b7525dca80caf7003e1 by Miss Islington (bot) in branch '3.8':
bpo-40135: Fix multiprocessing test_shared_memory_across_processes() (GH-19892)
https://github.com/python/cpython/commit/70fe95cdc9ac1b00d4f86b7525dca80caf7003e1
History
Date User Action Args
2020-05-04 15:25:29miss-islingtonsetmessages: + msg368059
2020-05-04 15:07:29vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg368057

stage: patch review -> resolved
2020-05-04 15:06:28miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request19217
2020-05-04 15:06:02vstinnersetmessages: + msg368056
2020-05-04 03:14:49shihai1991setmessages: + msg368007
2020-05-04 02:58:55shihai1991setkeywords: + patch
nosy: + shihai1991

pull_requests: + pull_request19203
stage: patch review
2020-04-01 13:21:33vstinnersetmessages: + msg365463
versions: - Python 3.7
2020-04-01 13:18:18vstinnercreate