This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: The Modules/ directory should not be added to sys.path
Type: Stage: resolved
Components: Versions: Python 3.4, Python 3.5
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Shared modules built with Modules/Setup are not found when run from build directory
View: 17095
Assigned To: Nosy List: Arfrever, loewis, ned.deily, pitrou, skrah, twouters, vstinner
Priority: normal Keywords: patch

Created on 2014-08-27 10:07 by vstinner, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
getpath_no_modules.patch vstinner, 2014-08-27 10:07 review
Messages (9)
msg225955 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-08-27 10:07
When Python is built from source, the "Modules/" subdirectory is added to sys.path on UNIX. I don't understand why: it does not contain .py files nor .so dynamic modules. Dynamic modules are built in "build/lib.linux-x86_64-3.5-pydebug".

A side effect of adding Modules/ subdirectory to sys.path is that Modules subdirectories (ex: _sqlite, _ctypes, _io, _decimal, etc.) can be used as packages.

For example, when the _decimal module cannot be compiled, Lib/decimal.py uses Modules/_decimal as a Python package which is wrong. The decimal becomes an empty module (no symbol, because Modules/_decimal does not contain an __init__.py nor any .so file) because decimal.py uses "import _decimal" at the end (see decimal.py).

Attached patch removes Modules/ from sys.path on UNIX.

Maybe adding Modules/ to sys.path was useful before the introduction of pybuildir.txt? See issue #9589 and the changeset 4742e7aea2f5 (and the issue #586680).
msg225957 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-08-27 10:17
A side effect of this issue is that when the _decimal cannot be build (ex: #22283), the Python implementation of the decimal cannot be used. Extract of buildbot test logs related to #22283:
---
Failed to build these modules:
_decimal

(...)
  File "/usr/home/buildbot/buildarea/3.x.krah-freebsd/build/Lib/test/test_statistics.py", line 15, in <module>
    from decimal import Decimal
ImportError: cannot import name 'Decimal'
---
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%203.x/builds/7213/steps/test/logs/stdio
msg225959 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-08-27 10:18
See also issue #22280 for the case of the wrong _decimal package.
msg225961 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-08-27 10:29
The motivation for this feature is that modules built as shared libraries through Modules/Setup end up in Modules, so Modules is added so that they are found.

I'd like to preserve support for building dynamic extension modules through Modules/Setup, but they don't have to live in Modules; putting them into build (say) would be fine as well.
msg225974 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-08-27 11:03
> I'd like to preserve support for building dynamic extension modules through Modules/Setup, but they don't have to live in Modules; putting them into build (say) would be fine as well.

Does it mean that you are ok to commit my patch?
msg225975 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2014-08-27 11:13
I think we have this behavior since 6c468df214dc and 227ce85bdbe0 (#17095).
msg225982 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-08-27 13:02
Another bug linked to the bad Modules/_ctypes package:
http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/8549/steps/test/logs/stdio

test test_ctypes crashed -- Traceback (most recent call last):
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/regrtest.py", line 1271, in runtest_inner
    the_module = importlib.import_module(abstest)
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2203, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2186, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2175, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1149, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1420, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/test_ctypes.py", line 3, in <module>
    from ctypes.test import load_tests
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ImportError: cannot import name 'Union'
msg225991 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-08-27 15:18
> Does it mean that you are ok to commit my patch?

No, because it will break #17095 again. A proper patch would do what you do, plus find some other solution to #17095. Just reverting 6c468df214dc is not ok.
msg226549 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-09-08 00:25
Closing as a duplicate of the re-opened Issue17095 (since the changes there haven't been released yet)
History
Date User Action Args
2022-04-11 14:58:07adminsetgithub: 66481
2014-09-08 00:25:40ned.deilysetstatus: open -> closed
versions: + Python 3.4, Python 3.5
superseder: Shared modules built with Modules/Setup are not found when run from build directory
messages: + msg226549

resolution: duplicate
stage: resolved
2014-08-31 10:00:05Arfreversetnosy: + Arfrever
2014-08-27 15:18:29loewissetmessages: + msg225991
2014-08-27 13:02:23vstinnersetmessages: + msg225982
2014-08-27 11:13:06skrahsetnosy: + twouters, ned.deily
messages: + msg225975
2014-08-27 11:03:29vstinnersetmessages: + msg225974
2014-08-27 10:29:52loewissetnosy: + loewis
messages: + msg225961
2014-08-27 10:25:32skrahlinkissue22280 superseder
2014-08-27 10:18:45vstinnersetmessages: + msg225959
2014-08-27 10:17:33vstinnersetnosy: + skrah
messages: + msg225957
2014-08-27 10:07:47vstinnercreate