This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Use dlopen() to load extensions on Darwin, where possible
Type: Stage:
Components: Build Versions: Python 2.5
process
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: anthonybaxter Nosy List: anthonybaxter, barnert, nnorwitz, ronaldoussoren, zpincus
Priority: normal Keywords: patch

Created on 2006-03-20 19:41 by zpincus, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
darwin_use_dlopen.patch zpincus, 2006-03-20 19:41 Patch file containing patches described above.
darwin_use_dlopen2.patch zpincus, 2006-03-22 08:34 Modified patch file -- won't match 'Darwin/10'
Messages (15)
msg49791 - (view) Author: Zach Pincus (zpincus) Date: 2006-03-20 19:41
Here is a patch to make Python on OS X 10.3 and above use dlopen() 
(via dynload_shlib.c) to load extension modules, and to make the dl 
module avaliable. 

The patch was generated against the SVN head as of yesterday, but can 
be applied (and I have done so with success) to Python 2.4.2.

RATIONALE
---------
On most unix-like operating systems, Python uses the dlopen() call to 
load extension modules. In addition, the way that these modules are 
opened can be modified via 'sys.setdlopenflags(...)'.

Modifications of how extensions are loaded are useful for several 
reasons (enough so that the standard Python docs (e.g. http://
docs.python.org/lib/module-sys.html ). In particular, if multiple 
modules need to share symbols, a call to sys.setdlopenflags is 
necessary.

Symbol sharing is especially important for interoperability of modules 
which wrap C++ classes, because GCC creates classes that resolve 
their run-time type identification by symbol identity. Thus, symbols 
must be shared globally for many C++ features to work properly for 
objects passed between modules.

On OS X / Darwin, Python uses some NeXT-derived APIs to load 
modules. Though these APIs provide analogues to the dlopenflags 
used to control how dlopen() loads modules, this interface is *not* 
exposed to the Python interpreter. Worse, sys.setdlopenflags remains 
available on Darwin, though calls to it are never heeded.

Fortunately, on OS X 10.3 and above, Apple has included dlopen as a 
standard function. In 10.3, this call is provided by a compatibility API; 
in 10.4, the dlopen() call is written to interface directly with the library 
loading mechanism and is now the recommended method for libraries 
to be opened for non Carbon/Cocoa tools.

IMPLEMENTATION
--------------
This (trivial) patch instructs the Python build process to use 
dynload_shlib.c (which uses dlopen) instead of dynload_next.c (which 
uses the NeXT-derived APIs). It also allows for the dl module to be built 
in order to provide access to the proper values for the various dlopen 
flags.

TESTING
-------
This patch can be configured and built into executables that build and 
test correctly on 10.3 and 10.4.

Because Python 2.5 and 2.4 do not currently compile properly on OS X 
10.2, I have not built or tested this patch on that OS version. However, 
the configure and compile process does select the appropriate 
dynload_next.c file to use, and compiles that correctly before breaking 
elsewhere. Thus, if the other errors are fixed for 10.2, these patches 
will work fine. (This is because they only change Python's behavior for 
10.3 and up.)

PATCHES
-------
There are three main components to the attached patch.

The first is a patch the 'configure.in' file to use dynload_shlib.c when it 
can, and a patch to the 'configure' file to sync it up with 'configure.in'. 

The second is a minor change to 'setup.py' and the dl module test to 
allow the dl module to be built and tested on OS X systems where dlfcn 
is available. (10.3 and above.)

The last part of the patchfile should be considered optional. This patch 
applies to 'Lib/test/regrtest.py', and it tells the testing suite that the dl 
test is not expected to be skipped anymore.

This is optional because if Python is ever built on 10.2, the test script 
will expect dl to work, when it only works on 10.3 and above. However, 
if Python on 10.2 is officially not supported, then this change should 
be made to properly test the dl functionality on all supported OS X 
platforms.

msg49792 - (view) Author: Andrew Barnert (barnert) Date: 2006-03-22 02:31
Logged In: YES 
user_id=1473180

One minor issue: Your pattern will match Darwin 10.0, which will come out in 
a few years as part of OS X 10.7, by which time nobody will remember why 
this script is written this way. Since it's easy to handle now, we might as well 
do so. On a side note, there are no 2.x-4.x versions (1.5 was renamed 5.1), 
so anyone who matches Darwin/2.* will be a false match.

Here's what I'd suggest:

+       Darwin/[0156]\..*) DYNLOADFILE="dynload_next.o";;

Meanwhile, your patch seems to work for me on an Intel iMac, but I haven't 
tried anything too sophisticated.
msg49793 - (view) Author: Zach Pincus (zpincus) Date: 2006-03-22 08:34
Logged In: YES 
user_id=748718

I just uploaded a new patch which addresses this issue, and fixes this same 
problem in a few other locations in the configure script.
msg49794 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2006-03-27 17:25
Logged In: YES 
user_id=580910

I have a number of questions. Firstly, how does symbol sharing between 
extensions work on windows? And more importantly, have you checked if this 
will break existing python code?

msg49795 - (view) Author: Zach Pincus (zpincus) Date: 2006-03-27 18:40
Logged In: YES 
user_id=748718

Ronald -
I am unfortunately not an expert on windows DLL loading. I do know that in 
non-GCC C++ implementations, it becomes far less critical to share symbols 
globally in many circumstances, because non-GCC C++ implementations 
tend to use string comparisons to resolve C++ object identity, and not use 
exact address comparisons on the typeinfo objects. Thus, these objects do 
not need to be shared between windows DLLs.

Beyond that, I do not know whether symbols from DLLs are loaded with 
global visibility in windows or not. Certainly the windows dlopen() equivalent 
*does not* support mode options. I presume anything tagged with _declspec
(dllexport) is loaded globally when the DLL is opened, but I am not sure.

However, that issue is largely tangential to the question of whether Python on 
OS X should work like Python on every other Unix-like system...

As to your other question -- have I checked if this will break existing python 
code -- the answer is of course yes. The python regression suite test passes 
on both 10.3 and 10.4 with this modification (as I mentioned), so that's a 
good indication that run-of-the-mill code will not be affected.

In fact, on 10.3, the dlopen() call is implemented under the hood with the 
exact same calls that Python's dynload_next.c file uses. In 10.4 this was 
changed so that both APIs talk to the same underlying system. Thus, nothing 
is really changing: both are functionally identical ways of getting the same OS 
X runtime loader (dyld) to load a bundle from disk. The only difference is that 
by using dlopen(), there are hooks in Python for manipulating the mode that 
is used to load that bundle from disk.

So, I cannot imagine a situation where this change would break code on the 
mac. While argument from failure of imagination is typically weak, this 
combined with the fact that the python tests pass without difficulty (including 
the dl module test), I am fairy confident in this patch. Moreover, I have been 
using this patch on my own computer for some time and have not run into 
any difficulties, even when loading fairly complex custom extensions.

Finally, this patch *only* modifies the codepath on Darwin/OS X, so there's 
zero possibility that it would affect anything on other platforms.
msg49796 - (view) Author: Zach Pincus (zpincus) Date: 2006-04-03 14:43
Logged In: YES 
user_id=748718

Another point --

The functions used in dynload_next.c, which is what Python
currently uses to load extensions on OS X, are officially
'discouraged' by Apple, in favor of the dlopen() functions:

http://developer.apple.com/documentation/DeveloperTools/Reference/
MachOReference/index.html

Here's dynload_next.c, for reference.
http://svn.python.org/view/python/trunk/Python/dynload_next.c?view=auto

Thus, it rather makes sense to move Python from this old and
in all probability soon-to-be-deprecated code path.
msg49797 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2006-04-03 17:30
Logged In: YES 
user_id=580910

I'm not too confident that running the testsuite would uncover problems with 
this patch, but the fact that you're running with this patch without running into 
problems indicate that this patch should be save.

I'm in favor of applying this patch (all of it) as this helps a usecase that comes 
up regularly on the pythonmac-sig mailinglist and seems to be safe. This patch 
isn't applied dynload_next should IMO be taught about sys.setdlopenflags.
msg49798 - (view) Author: Zach Pincus (zpincus) Date: 2006-04-04 04:36
Logged In: YES 
user_id=748718

The test suite does load in a plain-vanilla fashion a
bunch of C python extensions, so that's some 
evidence that the basic stuff works. Better, the dl 
tests also works, so direct usage of dlopen seems 
OK.

As you point out, the best test is usage. All I can 
say here is that my use of this patch works OK, 
even with exotic modules which require setting
the dlopenflags. I've asked on python-dev if 
anyone can think of some better test cases.

Now, if this patch is deemed too radical, I agree
that it would be a good idea to teach 
dynload_next.c about the dlopenflags. This
would be a bit odd in the sense that the flags
on the mac would be interpreted not as 
'dlopenflags' with values from dlfcn.h, but as
nsmoduleflags, with values from mach-o/dyld.h.
However, this would be better than the 
current state of affairs.

Finally, if this patch isn't added, we should at
least fix the rest of the configure.in script in 
the manner suggested by the first comment,
so that other things don't start randomly
breaking when Darwin 10 comes around...
msg49799 - (view) Author: Zach Pincus (zpincus) Date: 2006-04-07 15:46
Logged In: YES 
user_id=748718

OK, so I've done all of the tests suggested on Python-Dev
and things look fine. Bob gave his approval of this patch
both on Python-Dev and earlier on PythonMac-SIG, so it
appears that this patch is ready to go.
msg49800 - (view) Author: Anthony Baxter (anthonybaxter) (Python triager) Date: 2006-04-09 12:18
Logged In: YES 
user_id=29957

I'll land this patch shortly.
msg49801 - (view) Author: Anthony Baxter (anthonybaxter) (Python triager) Date: 2006-04-09 15:08
Logged In: YES 
user_id=29957

Checked in as r43748. I tested it on my OSX 10.3 box, seems
to work ok.
msg49802 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2006-04-09 19:54
Logged In: YES 
user_id=33168

I have a question about the configure stanzas changed. 
There are 3 changes like this:

Darwin/@<:@01567@:>@\..*)

Except that the last one only goes through 6, not 7.  Are
they meant to be inconsistent?  Perhaps a comment should be
added to explain this if it's correct.
msg49803 - (view) Author: Zach Pincus (zpincus) Date: 2006-04-09 20:25
Logged In: YES 
user_id=748718

The stanzas that look like this (there are only two):
-  Darwin/@<:@01234567@:>@.*) 
+  Darwin/@<:@01567@:>@\..*) 
were added in response to the first comment about
not matching Darwin/10, and the absence of any
Darwin versions 2-4. These changes are actually
unrelated to the dynload_shlib.c issue, and should
be viewed as simply cleanup. Note that that part
of the patch doesn't change any behavior, it
just makes the regex better behaved.

The only stanza that changes any behavior is the
following:
-	Darwin/*) DYNLOADFILE="dynload_next.o";;
+	# Use dynload_next.c only on 10.2 and below, which don't have native 
dlopen()
+	Darwin/@<:@0156@:>@\..*) DYNLOADFILE="dynload_next.o";;
which is pretty self-explanatory.

Basically, the top two stanzas will match any
Darwin version < 10.4, whereas my change 
matches any version < 10.3. This just reflects
the version of Darwin that Apple added various
pieces of "standard" *nix functionality so that
Python no longer needed to work around them.

Hopefully that's all clear! Let me know if there
are further questions, and thanks for reviewing
the patch.
msg49804 - (view) Author: Zach Pincus (zpincus) Date: 2006-04-09 20:36
Logged In: YES 
user_id=748718

The stanzas that look like this (there are only two):
-  Darwin/@<:@01234567@:>@.*) 
+  Darwin/@<:@01567@:>@\..*) 
were added in response to the first comment about
not matching Darwin/10, and the absence of any
Darwin versions 2-4. These changes are actually
unrelated to the dynload_shlib.c issue, and should
be viewed as simply cleanup. Note that that part
of the patch doesn't change any behavior, it
just makes the regex better behaved.

The only stanza that changes any behavior is the
following:
-	Darwin/*) DYNLOADFILE="dynload_next.o";;
+	# Use dynload_next.c only on 10.2 and below, which don't have native 
dlopen()
+	Darwin/@<:@0156@:>@\..*) DYNLOADFILE="dynload_next.o";;
which is pretty self-explanatory.

Basically, the top two stanzas will match any
Darwin version < 10.4, whereas my change 
matches any version < 10.3. This just reflects
the version of Darwin that Apple added various
pieces of "standard" *nix functionality so that
Python no longer needed to work around them.

Hopefully that's all clear! Let me know if there
are further questions, and thanks for reviewing
the patch.
msg49805 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2006-04-09 20:37
Logged In: YES 
user_id=33168

Oh, I think I understand why I'm confused--this is confusing
crap.  Can we put a table that maps the names appropriately
for those of us that don't know these things?  Something
like this (I don't know if it's correct, but you get the idea):

Darwin/0\..*  OS X 10.0
Darwin/1\..*  OS X 10.1
Darwin/2\..*  Does not exist
Darwin/3\..*  Does not exist
Darwin/4\..*  Does not exist
Darwin/5\..*  OS X 10.2
Darwin/6\..*  OS X 10.3
Darwin/7\..*  OS X 10.4
History
Date User Action Args
2022-04-11 14:56:16adminsetgithub: 43064
2009-03-30 05:19:00ajaksu2linkissue1516897 dependencies
2006-03-20 19:41:13zpincuscreate