This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: Shared modules built with Modules/Setup are not found when run from build directory
Type: Stage: needs patch
Components: Build Versions: Python 3.4, Python 3.5
Status: open Resolution:
Dependencies: Superseder:
Assigned To: ned.deily Nosy List: Arfrever, John.Malmberg, larry, loewis, matrixise, ned.deily, python-dev, twouters, vstinner
Priority: low Keywords: patch

Created on 2013-01-31 21:54 by twouters, last changed 2022-04-11 14:57 by admin.

File name Uploaded Description Edit
getpath.diff twouters, 2013-01-31 21:54 review
issue17095_modules_setup.patch ned.deily, 2014-05-25 03:30 review
Messages (12)
msg181048 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2013-01-31 21:54
At some point (probably in 3.2) the support for shared modules built using Modules/Setup was broken, for two reasons:

 - Python no longer considers '' when looking for a module called 'foo', but Modules/makesetup still appends '' for a 'foo' module.

 - Python no longer considers Modules/ to be the exec_prefix (and instead uses what was loaded from the build-info .txt written by but the build process still leaves these Modules/Setup-built shared modules in the Modules directory (and it doesn't know about the directory will make, so it can't really do anything else.)

This patch fixes both problems, by making Modules/makesetup name shared library modules '' and by adding the Modules subdirectory to sys.path (after the directory) when running from the build directory. (The existing build process has no problem _installing_ the Modules/Setup-built extension modules, it's just Python that doesn't know how to find them.)
msg219070 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-05-25 03:30
Your original patch LGTM.  However, since it was written, Setup *shared* support became even more broken due to changes in Makefile macros.  Attached is an updated patch.  I've never tried customizing Setup before so I don't have a lot of confidence that it now works as expected in all cases, but I tried a few cases (like building _ssl with a non-standard OpenSSL on OS X) and they did work.
msg219554 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-06-02 08:16
New changeset 6c468df214dc by Ned Deily in branch '3.4':
Issue #17095: Fix Modules/Setup *shared* support.

New changeset 227ce85bdbe0 by Ned Deily in branch 'default':
Issue #17095: Fix Modules/Setup *shared* support.
msg219555 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-06-02 08:19
Committed for release in 3.4.2 and 3.5.0.
msg225978 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-08-27 12:09
I reopen the issue because these changes broke the decimal module when the _decimal module is missing: see the issue #22285.

Martin von Loewis proposed to build modules in the build directory, not in the Modules directory directly:
msg226548 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-09-08 00:22
Clearly this breaks running from the build directory for various modules when their optional C extension module build fails.  I'll either change the Setup shared builds to use a build directory or revert the path changes prior to the upcoming 3.4.2.
msg226894 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-09-15 02:21
New changeset 781454f792c4 by Ned Deily in branch '3.4':
Issue #17095: Temporarily revert getpath.c change that added the Modules

New changeset d3939f602e1f by Ned Deily in branch 'default':
Issue #17095: merge from 3.4
msg229421 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2014-10-15 09:16
Hi Ned,

What are the news for this issue?
msg229490 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-10-15 18:29
Stéphane, as I understand it, this issue covers two problems:
(1) Building and installing shared modules using Modules/Setup was broken for current Python 3 releases;
(2) Using those shared modules when running Python from a build directory (rather than from a normal installed Python) is broken.

Problem (1) is now fixed.  Problem (2) remains to be properly fixed; it's a little tedious to get right as a fix needs to take into account the SO_ABI and ensure built modules get properly cleaned and all the right Makefile dependencies work.  I'd be happy to review a patch if someone gets to it before I do.  But it seems to me that (1) was the serious problem; (2) is more of a minor annoyance for developers with easy workarounds, e.g. use PYTHONPATH or install Python to a temporary location for testing.
msg229637 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2014-10-18 08:38
Hi Ned,

For the first point, maybe close this issue and open a new bug for the second.

What do you think?
msg229638 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-10-18 08:59
Sorry, I don't see the point of opening another issue, especially since most of the discussion in this issue has been about the second problem.  I have updated the issue title to be more specific, though.
msg229639 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2014-10-18 09:09
Thanks Ned
Date User Action Args
2022-04-11 14:57:41adminsetgithub: 61297
2014-10-18 09:09:12matrixisesetmessages: + msg229639
2014-10-18 08:59:31ned.deilysetmessages: + msg229638
title: Modules/Setup *shared* support broken -> Shared modules built with Modules/Setup are not found when run from build directory
2014-10-18 08:38:32matrixisesetmessages: + msg229637
2014-10-15 18:29:27ned.deilysetpriority: normal -> low

messages: + msg229490
2014-10-15 09:16:00matrixisesetnosy: + matrixise
messages: + msg229421
2014-10-14 16:11:57skrahsetnosy: - skrah
2014-09-15 02:25:34ned.deilysetpriority: release blocker -> normal
2014-09-15 02:21:32python-devsetmessages: + msg226894
2014-09-08 02:41:40John.Malmbergsetnosy: + John.Malmberg
2014-09-08 00:28:28ned.deilylinkissue22212 superseder
2014-09-08 00:25:40ned.deilylinkissue22285 superseder
2014-09-08 00:22:18ned.deilysetpriority: normal -> release blocker

nosy: + larry
messages: + msg226548

assignee: ned.deily
stage: resolved -> needs patch
2014-08-31 09:59:30Arfreversetnosy: + Arfrever
2014-08-27 12:09:59vstinnersetnosy: + loewis, skrah
2014-08-27 12:09:50vstinnersetstatus: closed -> open

nosy: + vstinner
messages: + msg225978

resolution: fixed ->
2014-06-02 08:19:48ned.deilysetstatus: open -> closed
resolution: fixed
messages: + msg219555

stage: patch review -> resolved
2014-06-02 08:16:35python-devsetnosy: + python-dev
messages: + msg219554
2014-05-25 03:30:15ned.deilysetfiles: + issue17095_modules_setup.patch
versions: + Python 3.5, - Python 3.3
nosy: + ned.deily

messages: + msg219070

stage: patch review
2013-10-14 14:15:01georg.brandlsetversions: - Python 3.2
2013-01-31 21:54:56twouterscreate