https://emptysqua.re/blog/getaddrinfo-deadlock/ claims that getaddrinfo may deadlock when using threads and fork on (amongst others) OSX due to using a global lock.

That lock is used when getaddrinfo is believed to be not thread safe, see the relevant code below (from the blog post):

/* On systems on which getaddrinfo() is believed to not be thread-safe,
   (this includes the getaddrinfo emulation) protect access with a lock. */
#if defined(WITH_THREAD) && (defined(__APPLE__) || \
    (defined(__FreeBSD__) && __FreeBSD_version+0 < 503000) || \
    defined(__OpenBSD__) || defined(__NetBSD__) || \
    defined(__VMS) || !defined(HAVE_GETADDRINFO))
#define USE_GETADDRINFO_LOCK
#endif

I think it is worthwhile to investigate whether or not getaddrinfo on OSX is really not thread safe. 

Some source code for OSX can be found at the link below, I haven't checked yet which OSX release this corresponds to:

http://www.opensource.apple.com/source/Libinfo/Libinfo-278/lookup.subproj/getaddrinfo.c

In Apple's Libinfo version 222.4.12 (corresponding to the last OS X 10.4 release), the man page says getaddrinfo isn't thread-safe:

http://www.opensource.apple.com/source/Libinfo/Libinfo-222.4.12/lookup.subproj/getaddrinfo.3

And here's its source:

http://www.opensource.apple.com/source/Libinfo/Libinfo-222.4.12/lookup.subproj/getaddrinfo.c

Glancing at the source naïvely, I might see the data race: getaddrinfo calls gai_lookupd, which reads and writes the global static variable "gai_proc". I can't see what will go wrong as a result of the race, but it sure LOOKS bad.

In the next release, version 278 (OS X 10.5.0), the thread-safety warning is gone from the man page:

http://www.opensource.apple.com/source/Libinfo/Libinfo-278/lookup.subproj/getaddrinfo.3

And getaddrinfo is largely rewritten:

http://www.opensource.apple.com/source/Libinfo/Libinfo-278/lookup.subproj/getaddrinfo.c

It calls a new function, "ds_getaddrinfo". But ds_getaddrinfo still accesses the global static variable "gai_proc"; I wonder why this is considered thread-safe now?

Related to #1288833, when FreeBSD 5.3's getaddrinfo was declared thread-safe.

NetBSD has a concurrency test for getaddrinfo, so I tried it on Mac to see if getaddrinfo is thread-safe here, too. It appears that it is thread-safe.

I copied NetBSD's getaddrinfo concurrency test "h_resolv.c" and the test's data file "d_mach" from src/tests/lib/libpthread in the NetBSD 7 source:

http://cvsweb.netbsd.org/bsdweb.cgi/src/tests/lib/libpthread/

I've also attached these files to this ticket.*

The test program h_resolv.c compiles fine on Mac OS X 10.10 with clang 7, and the test passes using the same parameters that the NetBSD test uses, resolving five host names on five threads:

./h_resolv -d -n 5 -h 5 d_mach

There's even evidence that getaddrinfo is actually concurrent on my Mac: resolving 100 hostnames on 1 thread takes about 6 seconds, but using 100 threads take only 3 seconds. To test concurrency I ran this:

sudo dscacheutil -flushcache
sudo killall -HUP mDNSResponder
time h_resolv -d -h 100 -n 1 d_mach

sudo dscacheutil -flushcache
sudo killall -HUP mDNSResponder
time h_resolv -d -h 1 -n 100 d_mach

Those sudo commands flush the Mac's DNS cache between runs.

* I removed one unresolvable domain, "cnftp.bjpu.edu.cn", from the test file d_mach. That domain took 15 seconds to time out and made the rest of the timing irrelevant.

An Apple Developer Relations engineer tells me it's "reasonable to assume that getaddrinfo() is thread safe" on OS X 10.5 and later. (He mentioned that iOS inherited the OS X 10.5 DNS architecture, so Apple phones, watches, TVs, hairdryers, etc. can all do concurrent DNS too.)

Before OS X 10.5 the DNS system was a mix of new Mac OS X features, mDNSResponder and Open Directory, along with historical lookupd from NeXT and libresolv from an old BSD. In 10.5 the system was cleaned up to depend on OS X's mDNSResponder consistently; in the process getaddrinfo became thread-safe.

He cites as further evidence of its thread safety:

* The "bugs" section of getaddrinfo's man page was removed in 10.5
* 10.5 is the first OS X to be UNIX '03 certified, which includes the requirement that getaddrinfo be thread-safe
* The system itself uses getaddrinfo extensively as if it's thread-safe
* He hasn't seen reports of thread-safety problems with getaddrinfo

He explained how the code works. In the latest version:

http://www.opensource.apple.com/source/Libinfo/Libinfo-476/lookup.subproj/libinfo.c

getaddrinfo calls down to libinfo’s "mdns" module:

http://www.opensource.apple.com/source/Libinfo/Libinfo-476/lookup.subproj/mdns_module.c

"mdns" uses the DNS-SD API declared in dns_sd.h, which he says is "well known to be thread safe." The DNS-SD API is part of the mDNSResponder project:

http://www.opensource.apple.com/source/mDNSResponder/mDNSResponder-576.30.4/

The key function is DNSServiceQueryRecord:

http://www.opensource.apple.com/source/mDNSResponder/mDNSResponder-576.30.4/mDNSShared/dnssd_clientstub.c

My contact concludes, "As you can see, it does an IPC over to the mDNSResponder process, at which point thread safety is assured."

I've created a Mac OS 10.4 virtual machine and reproduced the getaddrinfo concurrency bug there using the attached h_resolv.c. The man page on that OS version indeed includes the "not thread-safe" warning.

The same test passes on my Mac OS 10.10 system.

I am convinced that getaddrinfo is thread-safe on Mac OS 10.5+, and I'm attaching a patch to disable the lock on these systems.

Thanks for the thorough work! Hopefully we can apply this fix to 3.5, 3.6 and even 2.7.

Great, how do we ensure this gets merged soon?

Find an active developer who cares -- e.g. Martin Panter, Serhiy, Yury.

On Tue, Feb 2, 2016 at 5:46 AM, A. Jesse Jiryu Davis
<report@bugs.python.org> wrote:
>
> A. Jesse Jiryu Davis added the comment:
>
> Great, how do we ensure this gets merged soon?
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue25924>
> _______________________________________

Hi Jesse, could you please update your patch with a detailed comment summarizing your research?  Also it would be great if you can provide a separate patch for 2.7.

Second patch with a comment about how we know Mac OS 10.5+'s getaddrinfo is thread-safe. I'll wait for this to be merged before submitting another for Python 2.7.

Hello Jesse. Maybe Yury might be more qualified here than me to handle OS X stuff (correct me if I’m wrong). I mainly just work with Linux.

But reading around socketmodule.c, it sounds like we want to be able to build it on newer OS versions, but still have it compatible with older versions. If so, how does that MAC_OS_X_VERSION_10_5 macro work? Would some run-time check also be needed?

Thanks Martin. The MAC_OS_X_VERSION_10_5 macro ensures that Python is still compatible with Mac OS before version 10.5: if it's built on 10.4 or older, it locks around getaddrinfo (since it's not thread-safe there), and on 10.5 and later it doesn't lock.

I've built on a Mac OS 10.4 virtual machine and verified we'll still use the lock on that OS.

I think this is good enough, without a runtime check. We distribute prebuilt binaries for Mac OS 10.5+:

https://www.python.org/downloads/release/python-2711/

So those prebuilt Pythons should no longer lock around getaddrinfo. 

People who still use older Mac OSes (more than 12 years old!) will have to build Python themselves in order to get recent versions of Python, since we don't distribute binaries for them any more. Since they're building Python on old Mac OS, the MAC_OS_X_VERSION_10_5 macro will be undefined and they'll keep using the getaddrinfo lock.

Martin, here's a third patch, "try 3", which does a runtime version check instead of compile-time. It's a bit complex, compared to "try 2", which did a compile-time check instead.

I do NOT think this extra complexity is worth it to support Pythons that are built on 10.5+ and running on Mac OS 10.4 and older. 10.4 is long obsolete.

I propose merging "try 2" with the compile-time check. If that patch is accepted, I'll submit a patch for Python 2.7 also.

Thank you!

I agree that the approach in "try 2" is fine and the runtime check in "try 3" is overkill.  While it is possible to do so, we've never really supported building on an OS X release n for release m, where m < n, without using the m SDK on n (and in particular for m=10.4) in which case the "try 2" test should work correctly.  Ronald, what do you think?

I am happy to defer to Ned who probably knows a lot more than me about the OS X situation.

FTR it was the code added in 2006 by r45660 (Issue 1471925) that motivated my concern. Since that was added by Ronald, it would be good to hear his opinion :)

I think we have consensus for "try 2". I'm not a core dev, would one of you please merge 25924-getaddrinfo-is-thread-safe-2.patch? Thanks!

New changeset 58ebfa7c1361 by Ned Deily in branch '2.7':
Issue #25924: Avoid unnecessary serialization of getaddrinfo(3) calls on OS X
https://hg.python.org/cpython/rev/58ebfa7c1361

New changeset 86ddb4d747f8 by Ned Deily in branch '3.5':
Issue #25924: Avoid unnecessary serialization of getaddrinfo(3) calls on OS X
https://hg.python.org/cpython/rev/86ddb4d747f8

New changeset caca2b354773 by Ned Deily in branch 'default':
Issue #25924: merge with 3.5
https://hg.python.org/cpython/rev/caca2b354773

I've committed a revised version of the "try 2" patch for release in 2.7.12, 3.5.2, and 3.6.0.  The revisions were to better follow the somewhat arcane conventions of Apple's availability macros, in particular, to take into account the value of MACOSX_DEPLOYMENT_TARGET.  I didn't have an opportunity to test building it on an actual 10.4 (or 10.3 :) system but I did test with various settings of MACOSX_DEPLOYMENT_TARGET.  Thanks very much, Jesse, for the research and the patch and to everyone else for the reviews and comments.

Related to #26406, a fix for NetBSD and OpenBSD.

Here's a retelling of this bug report as a silly fantasy saga:

https://engineering.mongodb.com/post/the-saga-of-concurrent-dns-in-python-and-the-defeat-of-the-wicked-mutex-troll/