Author Paul Hobbs
Recipients Paul Hobbs
Date 2015-05-27.18:22:59
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1432750979.18.0.523128349285.issue24303@psf.upfronthosting.co.za>
In-reply-to
Content
Using pid namespacing it is possible to have multiple processes with the same pid.  "semlock_new" creates a semaphore file with the template "/dev/shm/mp{pid}-{counter}".  This can conflict if the same semaphore file already exists due to another Python process have the same pid.

This bug has been fixed in Python3: https://bugs.python.org/issue8713.  However, that patch is very large (40 files, ~4.4k changed lines) and only incidentally fixes this bug while introducing a large backwards-incompatible refactoring and feature addition.

The following small patch to just _multiprocessing/semaphore.c fixes the problem by using the system clock and retrying to avoid conflicts:

--- a/Modules/_multiprocessing/semaphore.c
+++ b/Modules/_multiprocessing/semaphore.c
@@ -7,6 +7,7 @@
  */
 
 #include "multiprocessing.h"
+#include <time.h>
 
 enum { RECURSIVE_MUTEX, SEMAPHORE };
 
@@ -419,7 +420,7 @@ semlock_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
 {
     char buffer[256];
     SEM_HANDLE handle = SEM_FAILED;
-    int kind, maxvalue, value;
+    int kind, maxvalue, value, try;
     PyObject *result;
     static char *kwlist[] = {"kind", "value", "maxvalue", NULL};
     static int counter = 0;
@@ -433,10 +434,24 @@ semlock_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
         return NULL;
     }
 
-    PyOS_snprintf(buffer, sizeof(buffer), "/mp%ld-%d", (long)getpid(), counter++);
+    /* With pid namespaces, we may have multiple processes with the same pid.
+     * Instead of relying on the pid to be unique, we use the microseconds time
+     * to attempt to a unique filename. */
+    for (try = 0; try < 100; ++try) {
+        struct timespec tv;
+        long arbitrary = clock_gettime(CLOCK_REALTIME, &tv) ? 0 : tv.tv_nsec;
+        PyOS_snprintf(buffer, sizeof(buffer), "/mp%ld-%d-%ld",
+                      (long)getpid(),
+                      counter++,
+                      arbitrary);
+        SEM_CLEAR_ERROR();
+        handle = SEM_CREATE(buffer, value, maxvalue);
+        if (handle != SEM_FAILED)
+            break;
+        else if (errno != EEXIST)
+            goto failure;
+    }
 
-    SEM_CLEAR_ERROR();
-    handle = SEM_CREATE(buffer, value, maxvalue);
     /* On Windows we should fail if GetLastError()==ERROR_ALREADY_EXISTS */
     if (handle == SEM_FAILED || SEM_GET_LAST_ERROR() != 0)
         goto failure;
History
Date User Action Args
2015-05-27 18:22:59Paul Hobbssetrecipients: + Paul Hobbs
2015-05-27 18:22:59Paul Hobbssetmessageid: <1432750979.18.0.523128349285.issue24303@psf.upfronthosting.co.za>
2015-05-27 18:22:59Paul Hobbslinkissue24303 messages
2015-05-27 18:22:59Paul Hobbscreate