classification
Title: investigate if getaddrinfo(3) on OSX is thread-safe
Type: enhancement Stage: resolved
Components: macOS Versions: Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ronaldoussoren Nosy List: emptysquare, gvanrossum, martin.panter, ned.deily, python-dev, ronaldoussoren, yselivanov
Priority: low Keywords: patch

Created on 2015-12-22 12:57 by ronaldoussoren, last changed 2016-11-17 12:27 by emptysquare. This issue is now closed.

Files
File name Uploaded Description Edit
h_resolv.c emptysquare, 2016-01-13 01:48 getaddrinfo concurrency test from NetBSD 7
d_mach emptysquare, 2016-01-13 01:48 getaddrinfo concurrency test data file from NetBSD 7
25924-getaddrinfo-is-thread-safe.patch emptysquare, 2016-01-30 03:18 Disable getaddrinfo lock on Mac OS 10.5+. review
25924-getaddrinfo-is-thread-safe-2.patch emptysquare, 2016-02-02 22:01 Disable getaddrinfo lock on Mac OS 10.5+, try 2. review
25924-getaddrinfo-is-thread-safe-3.patch emptysquare, 2016-02-10 02:45 Disable getaddrinfo lock on Mac OS 10.5+, try 3 (runtime check). review
Messages (21)
msg256838 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2015-12-22 12:57
https://emptysqua.re/blog/getaddrinfo-deadlock/ claims that getaddrinfo may deadlock when using threads and fork on (amongst others) OSX due to using a global lock.

That lock is used when getaddrinfo is believed to be not thread safe, see the relevant code below (from the blog post):

/* On systems on which getaddrinfo() is believed to not be thread-safe,
   (this includes the getaddrinfo emulation) protect access with a lock. */
#if defined(WITH_THREAD) && (defined(__APPLE__) || \
    (defined(__FreeBSD__) && __FreeBSD_version+0 < 503000) || \
    defined(__OpenBSD__) || defined(__NetBSD__) || \
    defined(__VMS) || !defined(HAVE_GETADDRINFO))
#define USE_GETADDRINFO_LOCK
#endif

I think it is worthwhile to investigate whether or not getaddrinfo on OSX is really not thread safe. 

Some source code for OSX can be found at the link below, I haven't checked yet which OSX release this corresponds to:

http://www.opensource.apple.com/source/Libinfo/Libinfo-278/lookup.subproj/getaddrinfo.c
msg257610 - (view) Author: A. Jesse Jiryu Davis (emptysquare) * Date: 2016-01-06 15:15
In Apple's Libinfo version 222.4.12 (corresponding to the last OS X 10.4 release), the man page says getaddrinfo isn't thread-safe:

http://www.opensource.apple.com/source/Libinfo/Libinfo-222.4.12/lookup.subproj/getaddrinfo.3

And here's its source:

http://www.opensource.apple.com/source/Libinfo/Libinfo-222.4.12/lookup.subproj/getaddrinfo.c

Glancing at the source naïvely, I might see the data race: getaddrinfo calls gai_lookupd, which reads and writes the global static variable "gai_proc". I can't see what will go wrong as a result of the race, but it sure LOOKS bad.

In the next release, version 278 (OS X 10.5.0), the thread-safety warning is gone from the man page:

http://www.opensource.apple.com/source/Libinfo/Libinfo-278/lookup.subproj/getaddrinfo.3

And getaddrinfo is largely rewritten:

http://www.opensource.apple.com/source/Libinfo/Libinfo-278/lookup.subproj/getaddrinfo.c

It calls a new function, "ds_getaddrinfo". But ds_getaddrinfo still accesses the global static variable "gai_proc"; I wonder why this is considered thread-safe now?
msg258121 - (view) Author: A. Jesse Jiryu Davis (emptysquare) * Date: 2016-01-12 23:40
Related to #1288833, when FreeBSD 5.3's getaddrinfo was declared thread-safe.
msg258125 - (view) Author: A. Jesse Jiryu Davis (emptysquare) * Date: 2016-01-13 01:51
NetBSD has a concurrency test for getaddrinfo, so I tried it on Mac to see if getaddrinfo is thread-safe here, too. It appears that it is thread-safe.

I copied NetBSD's getaddrinfo concurrency test "h_resolv.c" and the test's data file "d_mach" from src/tests/lib/libpthread in the NetBSD 7 source:

http://cvsweb.netbsd.org/bsdweb.cgi/src/tests/lib/libpthread/

I've also attached these files to this ticket.*

The test program h_resolv.c compiles fine on Mac OS X 10.10 with clang 7, and the test passes using the same parameters that the NetBSD test uses, resolving five host names on five threads:

./h_resolv -d -n 5 -h 5 d_mach

There's even evidence that getaddrinfo is actually concurrent on my Mac: resolving 100 hostnames on 1 thread takes about 6 seconds, but using 100 threads take only 3 seconds. To test concurrency I ran this:

sudo dscacheutil -flushcache
sudo killall -HUP mDNSResponder
time h_resolv -d -h 100 -n 1 d_mach

sudo dscacheutil -flushcache
sudo killall -HUP mDNSResponder
time h_resolv -d -h 1 -n 100 d_mach

Those sudo commands flush the Mac's DNS cache between runs.

* I removed one unresolvable domain, "cnftp.bjpu.edu.cn", from the test file d_mach. That domain took 15 seconds to time out and made the rest of the timing irrelevant.
msg258163 - (view) Author: A. Jesse Jiryu Davis (emptysquare) * Date: 2016-01-13 19:43
An Apple Developer Relations engineer tells me it's "reasonable to assume that getaddrinfo() is thread safe" on OS X 10.5 and later. (He mentioned that iOS inherited the OS X 10.5 DNS architecture, so Apple phones, watches, TVs, hairdryers, etc. can all do concurrent DNS too.)

Before OS X 10.5 the DNS system was a mix of new Mac OS X features, mDNSResponder and Open Directory, along with historical lookupd from NeXT and libresolv from an old BSD. In 10.5 the system was cleaned up to depend on OS X's mDNSResponder consistently; in the process getaddrinfo became thread-safe.

He cites as further evidence of its thread safety:

* The "bugs" section of getaddrinfo's man page was removed in 10.5
* 10.5 is the first OS X to be UNIX '03 certified, which includes the requirement that getaddrinfo be thread-safe
* The system itself uses getaddrinfo extensively as if it's thread-safe
* He hasn't seen reports of thread-safety problems with getaddrinfo

He explained how the code works. In the latest version:

http://www.opensource.apple.com/source/Libinfo/Libinfo-476/lookup.subproj/libinfo.c

getaddrinfo calls down to libinfo’s "mdns" module:

http://www.opensource.apple.com/source/Libinfo/Libinfo-476/lookup.subproj/mdns_module.c

"mdns" uses the DNS-SD API declared in dns_sd.h, which he says is "well known to be thread safe." The DNS-SD API is part of the mDNSResponder project:

http://www.opensource.apple.com/source/mDNSResponder/mDNSResponder-576.30.4/

The key function is DNSServiceQueryRecord:

http://www.opensource.apple.com/source/mDNSResponder/mDNSResponder-576.30.4/mDNSShared/dnssd_clientstub.c

My contact concludes, "As you can see, it does an IPC over to the mDNSResponder process, at which point thread safety is assured."
msg259233 - (view) Author: A. Jesse Jiryu Davis (emptysquare) * Date: 2016-01-30 03:18
I've created a Mac OS 10.4 virtual machine and reproduced the getaddrinfo concurrency bug there using the attached h_resolv.c. The man page on that OS version indeed includes the "not thread-safe" warning.

The same test passes on my Mac OS 10.10 system.

I am convinced that getaddrinfo is thread-safe on Mac OS 10.5+, and I'm attaching a patch to disable the lock on these systems.
msg259234 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-01-30 03:30
Thanks for the thorough work! Hopefully we can apply this fix to 3.5, 3.6 and even 2.7.
msg259387 - (view) Author: A. Jesse Jiryu Davis (emptysquare) * Date: 2016-02-02 13:46
Great, how do we ensure this gets merged soon?
msg259404 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-02-02 17:01
Find an active developer who cares -- e.g. Martin Panter, Serhiy, Yury.

On Tue, Feb 2, 2016 at 5:46 AM, A. Jesse Jiryu Davis
<report@bugs.python.org> wrote:
>
> A. Jesse Jiryu Davis added the comment:
>
> Great, how do we ensure this gets merged soon?
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue25924>
> _______________________________________
msg259424 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-02-02 19:27
Hi Jesse, could you please update your patch with a detailed comment summarizing your research?  Also it would be great if you can provide a separate patch for 2.7.
msg259437 - (view) Author: A. Jesse Jiryu Davis (emptysquare) * Date: 2016-02-02 22:01
Second patch with a comment about how we know Mac OS 10.5+'s getaddrinfo is thread-safe. I'll wait for this to be merged before submitting another for Python 2.7.
msg259457 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-02-03 05:39
Hello Jesse. Maybe Yury might be more qualified here than me to handle OS X stuff (correct me if I’m wrong). I mainly just work with Linux.

But reading around socketmodule.c, it sounds like we want to be able to build it on newer OS versions, but still have it compatible with older versions. If so, how does that MAC_OS_X_VERSION_10_5 macro work? Would some run-time check also be needed?
msg259480 - (view) Author: A. Jesse Jiryu Davis (emptysquare) * Date: 2016-02-03 13:46
Thanks Martin. The MAC_OS_X_VERSION_10_5 macro ensures that Python is still compatible with Mac OS before version 10.5: if it's built on 10.4 or older, it locks around getaddrinfo (since it's not thread-safe there), and on 10.5 and later it doesn't lock.

I've built on a Mac OS 10.4 virtual machine and verified we'll still use the lock on that OS.

I think this is good enough, without a runtime check. We distribute prebuilt binaries for Mac OS 10.5+:

https://www.python.org/downloads/release/python-2711/

So those prebuilt Pythons should no longer lock around getaddrinfo. 

People who still use older Mac OSes (more than 12 years old!) will have to build Python themselves in order to get recent versions of Python, since we don't distribute binaries for them any more. Since they're building Python on old Mac OS, the MAC_OS_X_VERSION_10_5 macro will be undefined and they'll keep using the getaddrinfo lock.
msg259967 - (view) Author: A. Jesse Jiryu Davis (emptysquare) * Date: 2016-02-10 02:45
Martin, here's a third patch, "try 3", which does a runtime version check instead of compile-time. It's a bit complex, compared to "try 2", which did a compile-time check instead.

I do NOT think this extra complexity is worth it to support Pythons that are built on 10.5+ and running on Mac OS 10.4 and older. 10.4 is long obsolete.

I propose merging "try 2" with the compile-time check. If that patch is accepted, I'll submit a patch for Python 2.7 also.

Thank you!
msg259970 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-02-10 03:25
I agree that the approach in "try 2" is fine and the runtime check in "try 3" is overkill.  While it is possible to do so, we've never really supported building on an OS X release n for release m, where m < n, without using the m SDK on n (and in particular for m=10.4) in which case the "try 2" test should work correctly.  Ronald, what do you think?
msg259974 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-02-10 04:11
I am happy to defer to Ned who probably knows a lot more than me about the OS X situation.

FTR it was the code added in 2006 by r45660 (Issue 1471925) that motivated my concern. Since that was added by Ronald, it would be good to hear his opinion :)
msg260242 - (view) Author: A. Jesse Jiryu Davis (emptysquare) * Date: 2016-02-13 16:09
I think we have consensus for "try 2". I'm not a core dev, would one of you please merge 25924-getaddrinfo-is-thread-safe-2.patch? Thanks!
msg260297 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-02-15 05:58
New changeset 58ebfa7c1361 by Ned Deily in branch '2.7':
Issue #25924: Avoid unnecessary serialization of getaddrinfo(3) calls on OS X
https://hg.python.org/cpython/rev/58ebfa7c1361

New changeset 86ddb4d747f8 by Ned Deily in branch '3.5':
Issue #25924: Avoid unnecessary serialization of getaddrinfo(3) calls on OS X
https://hg.python.org/cpython/rev/86ddb4d747f8

New changeset caca2b354773 by Ned Deily in branch 'default':
Issue #25924: merge with 3.5
https://hg.python.org/cpython/rev/caca2b354773
msg260299 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-02-15 06:37
I've committed a revised version of the "try 2" patch for release in 2.7.12, 3.5.2, and 3.6.0.  The revisions were to better follow the somewhat arcane conventions of Apple's availability macros, in particular, to take into account the value of MACOSX_DEPLOYMENT_TARGET.  I didn't have an opportunity to test building it on an actual 10.4 (or 10.3 :) system but I did test with various settings of MACOSX_DEPLOYMENT_TARGET.  Thanks very much, Jesse, for the research and the patch and to everyone else for the reviews and comments.
msg260731 - (view) Author: A. Jesse Jiryu Davis (emptysquare) * Date: 2016-02-23 13:27
Related to #26406, a fix for NetBSD and OpenBSD.
msg281036 - (view) Author: A. Jesse Jiryu Davis (emptysquare) * Date: 2016-11-17 12:27
Here's a retelling of this bug report as a silly fantasy saga:

https://engineering.mongodb.com/post/the-saga-of-concurrent-dns-in-python-and-the-defeat-of-the-wicked-mutex-troll/
History
Date User Action Args
2016-11-17 12:27:39emptysquaresetmessages: + msg281036
2016-02-23 13:27:52emptysquaresetmessages: + msg260731
2016-02-15 06:37:31ned.deilysetstatus: open -> closed
resolution: fixed
messages: + msg260299

stage: patch review -> resolved
2016-02-15 05:58:08python-devsetnosy: + python-dev
messages: + msg260297
2016-02-13 16:09:34emptysquaresetmessages: + msg260242
2016-02-10 04:11:21martin.pantersetmessages: + msg259974
2016-02-10 03:25:35ned.deilysetmessages: + msg259970
stage: patch review
2016-02-10 02:45:54emptysquaresetfiles: + 25924-getaddrinfo-is-thread-safe-3.patch

messages: + msg259967
2016-02-03 13:46:51emptysquaresetmessages: + msg259480
2016-02-03 05:39:01martin.pantersetnosy: + martin.panter
messages: + msg259457
2016-02-02 22:01:21emptysquaresetfiles: + 25924-getaddrinfo-is-thread-safe-2.patch

messages: + msg259437
2016-02-02 19:27:39yselivanovsetnosy: + yselivanov
messages: + msg259424
2016-02-02 17:01:53gvanrossumsetmessages: + msg259404
2016-02-02 13:46:17emptysquaresetmessages: + msg259387
2016-01-30 03:30:27gvanrossumsetnosy: + gvanrossum
messages: + msg259234
2016-01-30 03:18:08emptysquaresetfiles: + 25924-getaddrinfo-is-thread-safe.patch
keywords: + patch
messages: + msg259233
2016-01-13 19:43:42emptysquaresetmessages: + msg258163
2016-01-13 01:51:28emptysquaresetmessages: + msg258125
2016-01-13 01:48:35emptysquaresetfiles: + d_mach
2016-01-13 01:48:11emptysquaresetfiles: + h_resolv.c
2016-01-12 23:40:01emptysquaresetmessages: + msg258121
2016-01-06 15:15:53emptysquaresetnosy: + emptysquare
messages: + msg257610
2015-12-22 12:57:10ronaldoussorencreate