Title: Add ability to prefix posix semaphore names created by multiprocessing module
Type: enhancement Stage: resolved
Components: Extension Modules Versions: Python 2.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Marc.Liyanage, iritkatriel, jnoller, ned.deily, python-dev, ronaldoussoren, sbt
Priority: normal Keywords:

Created on 2013-11-01 19:53 by Marc.Liyanage, last changed 2020-12-01 15:25 by iritkatriel. This issue is now closed.

Messages (9)
msg201936 - (view) Author: Marc Liyanage (Marc.Liyanage) Date: 2013-11-01 19:53
I'm running Python 2.7 in a sandboxed OS X app. There are restrictions on the naming of POSIX semaphores when running in the sandbox. Specifically, there is a mandatory prefix.

I would like the ability to inject this prefix into the part of the multiprocessing module that creates the semaphore, perhaps by way of an environment variable.

I see that this is how the module generates the semaphore name:

    PyOS_snprintf(buffer, sizeof(buffer), "/mp%ld-%d", (long)getpid(), counter++);

Apple's documentation about the restrictions are here:

The relevant part:

"POSIX semaphores and shared memory names must begin with the application group identifier, followed by a slash (/), followed by a name of your choosing."

So if I could inject that string before the leasing slash, I could make this work.
msg201937 - (view) Author: Marc Liyanage (Marc.Liyanage) Date: 2013-11-01 19:54
"leasing" -> "leading"
msg201938 - (view) Author: Marc Liyanage (Marc.Liyanage) Date: 2013-11-01 20:23
I'm thinking something along the lines of

    if (env && *env != '\0') {
        PyOS_snprintf(buffer, sizeof(buffer), "%s/mp%ld-%d", env, (long)getpid(), counter++);
    } else {
        PyOS_snprintf(buffer, sizeof(buffer), "/mp%ld-%d", (long)getpid(), counter++);
msg201939 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-11-01 20:34
I don't have any experience using OS X sandboxing yet but I wonder whether there are other instances of semaphores or other resources used elsewhere in the standard library that would benefit from a common solution.
msg201940 - (view) Author: Marc Liyanage (Marc.Liyanage) Date: 2013-11-01 20:39
From the description above, I would guess shared memory names as well, but I don't know which parts of the Python library use those, if any.

In general, it's easy to adjust the Python code to deal with the restriction of the sandboxed environment (mostly file-system releated), and I haven't encountered any other issues that require changes to the C code.

The name generation for semaphores is one exception where it's not possible to work around it in pure Python code.
msg201941 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-11-01 20:54
Although it is undocumented, in python 3.4 you can control the prefix used by doing

    multiprocessing.current_process()._config['semprefix'] = 'myprefix'

in the main process at the beginning of the program.

Unfortunately, this will make the full prefix '/myprefix', so it will still start with '/'.  Changing this for 3.4 would be easy, but I don't know if it is a good idea to change 2.7.

Note that your suggested change can cause a buffer overflow.
msg201942 - (view) Author: Marc Liyanage (Marc.Liyanage) Date: 2013-11-01 21:13
Great to hear that this is mostly supported in 3.4. I would definitely like to see the small change added that lets me insert a string before the leading slash.

It would also be helpful to have it in Python 2.7 as well.
msg201982 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-11-02 17:09
New changeset 1b5506fc6a50 by Richard Oudkerk in branch 'default':
Issue #19478: Make choice of semaphore prefix more flexible.
msg382262 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-12-01 15:25
Fixed in 3.4, won't be fixed in 2.7.
Date User Action Args
2020-12-01 15:25:13iritkatrielsetstatus: open -> closed

nosy: + iritkatriel
messages: + msg382262

resolution: fixed
stage: resolved
2013-11-02 17:09:36python-devsetnosy: + python-dev
messages: + msg201982
2013-11-01 21:13:33Marc.Liyanagesetmessages: + msg201942
2013-11-01 20:54:27sbtsetmessages: + msg201941
2013-11-01 20:39:09Marc.Liyanagesetmessages: + msg201940
2013-11-01 20:34:43ned.deilysetmessages: + msg201939
2013-11-01 20:26:14ned.deilysetnosy: + ronaldoussoren, ned.deily
2013-11-01 20:23:01Marc.Liyanagesetmessages: + msg201938
2013-11-01 19:54:33Marc.Liyanagesetmessages: + msg201937
2013-11-01 19:53:28Marc.Liyanagecreate