This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: TestSharedMemory fails on AMD64 FreeBSD CURRENT Shared 3.x
Type: resource usage Stage: resolved
Components: Tests Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: davin, eryksun, giampaolo.rodola, pitrou, vstinner
Priority: normal Keywords: patch

Created on 2019-02-25 09:03 by vstinner, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 12036 merged davin, 2019-02-25 20:18
Messages (9)
msg336503 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-25 09:03
TestSharedMemory fails on AMD64 FreeBSD CURRENT Shared 3.x:

* The tests should be skipped on this buildbot worker
* Or the feature should be fixed on FreeBSD

https://buildbot.python.org/all/#/builders/168/builds/617

Example:

ERROR: test_shared_memory_ShareableList_basics (test.test_multiprocessing_forkserver.WithProcessesTestSharedMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/_test_multiprocessing.py", line 3770, in test_shared_memory_ShareableList_basics
    sl = shared_memory.ShareableList(
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/shared_memory.py", line 305, in __init__
    self.shm = SharedMemory(name, create=True, size=requested_size)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/shared_memory.py", line 88, in __init__
    self._fd = _posixshmem.shm_open(
OSError: [Errno 22] Invalid argument: 'psm_f66c2bc7b9'

Failures:

ERROR: test_shared_memory_ShareableList_basics (test.test_multiprocessing_forkserver.WithProcessesTestSharedMemory)
ERROR: test_shared_memory_ShareableList_pickling (test.test_multiprocessing_forkserver.WithProcessesTestSharedMemory)
ERROR: test_shared_memory_SharedMemoryManager_basics (test.test_multiprocessing_forkserver.WithProcessesTestSharedMemory)
ERROR: test_shared_memory_across_processes (test.test_multiprocessing_forkserver.WithProcessesTestSharedMemory)
ERROR: test_shared_memory_basics (test.test_multiprocessing_forkserver.WithProcessesTestSharedMemory)

ERROR: test_shared_memory_ShareableList_basics (test.test_multiprocessing_fork.WithProcessesTestSharedMemory)
ERROR: test_shared_memory_ShareableList_pickling (test.test_multiprocessing_fork.WithProcessesTestSharedMemory)
ERROR: test_shared_memory_SharedMemoryManager_basics (test.test_multiprocessing_fork.WithProcessesTestSharedMemory)
ERROR: test_shared_memory_across_processes (test.test_multiprocessing_fork.WithProcessesTestSharedMemory)
ERROR: test_shared_memory_basics (test.test_multiprocessing_fork.WithProcessesTestSharedMemory)

ERROR: test_shared_memory_ShareableList_basics (test.test_multiprocessing_spawn.WithProcessesTestSharedMemory)
ERROR: test_shared_memory_ShareableList_pickling (test.test_multiprocessing_spawn.WithProcessesTestSharedMemory)
ERROR: test_shared_memory_SharedMemoryManager_basics (test.test_multiprocessing_spawn.WithProcessesTestSharedMemory)
ERROR: test_shared_memory_across_processes (test.test_multiprocessing_spawn.WithProcessesTestSharedMemory)
ERROR: test_shared_memory_basics (test.test_multiprocessing_spawn.WithProcessesTestSharedMemory)
msg336522 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2019-02-25 14:07
Though apparently undocumented on FreeBSD, their implementation of shm_open differs from others in the following way:  all names for shared memory blocks *must* begin with a slash.  This requirement does not exist on OpenBSD.

According to its man page on shm_open, FreeBSD does at least communicate the following non-standard, additional requirement:
    Two processes opening the same path
    are guaranteed to access the same shared memory object if and only if
    path begins with a slash (`/') character.


Given that this requirement is not universal and because a leading slash controls other behaviors on platforms like Windows, it would be confusing to make a leading slash a universal requirement.

Likewise, requiring users on FreeBSD to be aware of this nuance would be contrary to the goals of the SharedMemory class.


I will prepare a patch to prepend a leading slash onto the requested shared memory block name and detect the need for it.

I have verified that this does solve the problem on FreeBSD and that all tests then pass.  I will test NetBSD and DragonflyBSD to see if they also impose FreeBSD's undocumented requirement.
msg336531 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2019-02-25 15:08
In local testing, I found the following systems to impose the leading slash as a requirement for simply creating a shared memory block:
* NetBSD 8.0
* FreeBSD 12.x
* TrueOS 18.12 (the OS formerly known as PC-BSD)

I found the following systems to have no required leading slash and all tests currently pass without modification:
* OpenBSD 6.4
* DragonflyBSD 5.4
msg336553 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2019-02-25 21:04
I have locally tested GH-12036 on all 5 of the aforementioned OSes and all are made happy by the patch.

Victor:  If we want to go ahead and apply this patch right away to hopefully make the FreeBSD buildbot go green, the nature of this change is sufficiently small that it will still be easy to change during the alpha period.
msg336559 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-25 21:29
Apply the patch, we will see if the buildbot color changes :-)
msg336566 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2019-02-25 22:41
New changeset aadef2b41600cb6a4f845cdc4cea001c916d8745 by Davin Potts in branch 'master':
bpo-36102: Prepend slash to all POSIX shared memory block names (#12036)
https://github.com/python/cpython/commit/aadef2b41600cb6a4f845cdc4cea001c916d8745
msg336569 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2019-02-25 22:46
Closing.

Thank you Giampaolo for jumping in so quickly to review!

Thank you Victor for catching this on the buildbot.  Though what is this talk of "_if_ the color changes"? ;)
msg336571 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-25 23:13
FYI the test also failed on AMD64 FreeBSD 10-STABLE Non-Debug 3.x:
https://buildbot.python.org/all/#/builders/167/builds/600

I will keep on eye on this buildbot worker as well ;-)
msg336595 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-02-26 02:19
> Given that this requirement is not universal and because a leading 
> slash controls other behaviors on platforms like Windows

Forward slash has no special meaning anywhere in the names of NT objects, such as memory Section objects (aka file-mapping objects) and registry Key objects. Only the file API translates forward slash to backslash, and that's only implemented for the sake of DOS compatibility in a few core functions that normalize and translate DOS paths.

Maybe you're thinking of the "Global\\" and "Session\\" prefixes for named kernel objects, which require a backslash since the name is really a handle-relative NT path. The session's object directory contains object symbolic links named "Global" and "Session". A name such as "Global\spam" resolves to "spam" in the global object directory, but "Global/spam", with forward slash, is just a regular name in the local directory.
History
Date User Action Args
2022-04-11 14:59:11adminsetgithub: 80283
2019-02-26 02:19:14eryksunsetnosy: + eryksun
messages: + msg336595
2019-02-25 23:13:12vstinnersetmessages: + msg336571
2019-02-25 22:46:14davinsetstatus: open -> closed
resolution: fixed
messages: + msg336569

stage: patch review -> resolved
2019-02-25 22:41:54davinsetmessages: + msg336566
2019-02-25 22:37:48roufique7settype: resource usage
2019-02-25 21:29:05vstinnersetmessages: + msg336559
2019-02-25 21:06:02davinsetstage: patch review
2019-02-25 21:04:55davinsetmessages: + msg336553
stage: patch review -> (no value)
2019-02-25 20:18:42davinsetkeywords: + patch
stage: patch review
pull_requests: + pull_request12065
2019-02-25 15:08:03davinsetmessages: + msg336531
2019-02-25 14:07:18davinsetmessages: + msg336522
2019-02-25 09:03:37vstinnercreate