This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author asheesh
Recipients asheesh, asvetlov, yselivanov, zach.ware
Date 2020-05-19.21:07:44
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1589922464.68.0.792694033918.issue40692@roundup.psfhosted.org>
In-reply-to
Content
Some parts of test_concurrent_futures make sense to run even on Python builds where multiprocessing.synchronize is absent. At the moment, test_concurrent_futures.py skips all tests in the file if multiprocessing.synchronize is missing from the Python build. I have a patch to enable more tests, which I'll submit as a GitHub pull request.

The reason I want this patch is while working on CPython on Android, I saw that test_concurrent_futures refused to run its test suite, and I became very afraid that this meant that asyncio on CPython on Android was doomed. In fact, I was afraid for no reason. I want future people porting CPython to be less scared than I was. :)

I discovered that the only part of concurrent.futures that requires multiprocessing.synchronize is concurrent.futures.ProcessPoolExecutor.

concurrent.futures.ProcessPoolExecutor already has a way to check at runtime if it can work on the host system, so this patch leans on that.

This patch fixes a few other tests to skip themselves in the case that ProcessPoolExecutor cannot function.

I tested this by building CPython (with my patch) with adding a configure flag on macOS:

$ ./configure --with-pydebug --with-openssl=$(brew --prefix openssl) ac_cv_posix_semaphores_enabled=no

Why multiprocessing.synchronize is missing
------------------------------------------

multiprocessing.synchronize imports _multiprocessing.SemLock, which is based on a libc function called sem_open(), which implements named semaphores, which allows the creation of lock-esque counters that can be used from multiple processes. multiprocessing.c currently only creates the SemLock property if sem_open() is considered "working" through a check in ./configure; I've short-circuited that check above to be a "no".

On Android, sem_open() exists but simply returns ENOSYS; see its implementation here: https://android.googlesource.com/platform/external/pthreads/+/master/sem_open.c

Android isn't the only platform missing sem_open()

Past work
---------

issue3770 is an older conversation about a related issue. issue3770 has also attracted comments from someone who wants CPython's test suite to run to completion on their platform lacking sem_open() and therefore lacking multiprocessing.synchronize. This patch should solve their problem, I hope.

Future work
-----------

Automated testing: In order to prevent regressions, I'm interested in hosting a buildbot for at least one year, hopefully in perpetuity, on which we call ./configure with ac_cv_posix_semaphores_enabled=no.

I'd like to improve the error message when the user imports multiprocessing.synchronize so it points at some new documentation text that I (or someone) adds to CPython, rather than the issue number. I'm OK taking on the work of writing those docs in a follow-up issue if we merge this.

Thanks
------

Thanks to Zachary Ware for discussing this with me in an CPython contributor office hours time slot before I filed the issue. Thanks to Dr. Russell Keith-Magee, Glyph Lefkowitz, Nathaniel Smith, and Geoffrey Thomas for informal conversations and technical research.
History
Date User Action Args
2020-05-19 21:07:44asheeshsetrecipients: + asheesh, asvetlov, zach.ware, yselivanov
2020-05-19 21:07:44asheeshsetmessageid: <1589922464.68.0.792694033918.issue40692@roundup.psfhosted.org>
2020-05-19 21:07:44asheeshlinkissue40692 messages
2020-05-19 21:07:44asheeshcreate