classification
Title: pkgutil.extend_path do not recognize py{c,o} file
Type: behavior Stage: needs patch
Components: Library (Lib) Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Alexandre.Badez, barry, eric.araujo, eric.snow, ncoghlan, pje
Priority: normal Keywords: patch

Created on 2011-03-02 14:31 by Alexandre.Badez, last changed 2020-11-16 17:17 by iritkatriel.

Files
File name Uploaded Description Edit
patch.diff Alexandre.Badez, 2011-03-02 14:59 review
Messages (6)
msg129896 - (view) Author: Alexandre Badez (Alexandre.Badez) Date: 2011-03-02 14:31
extend_path only test if "init.py" files exist, but it should also test "init.pyc" and/or "init.pyo".
msg129898 - (view) Author: Alexandre Badez (Alexandre.Badez) Date: 2011-03-02 14:50
I've made a simple patch.
msg129900 - (view) Author: Alexandre Badez (Alexandre.Badez) Date: 2011-03-02 14:59
A little change in the patch
msg137453 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-01 18:17
Hi, thanks for the report and patch.  I think the code should not find byte-compiled files if sys.dont_write_bytecode is true, and it should not find pyo files when sys.flags.optimize has a certain value (I don’t remember if it’s 1 or 2).  It also requires tests.
msg148556 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-11-29 12:50
.pyo files are used if sys.dont_write_bytecode is false and sys.flags.optimize is >= 1 (IOW, true).

For Python 3.2 and 3.3, imp.cache_from_source should be used to get the right paths (see PEP 3147).

Before you put more work into this, it would be nice to get confirmation from one import expert that the bug is valid: I know the import system only superficially, and I’m not sure that package/__init__.pyc / .pyo is supported by import (if not, then pkgutil should also not support it).
msg148576 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-11-29 15:09
On Nov 29, 2011, at 12:50 PM, Éric Araujo wrote:

>Before you put more work into this, it would be nice to get confirmation from
>one import expert that the bug is valid: I know the import system only
>superficially, and I’m not sure that package/__init__.pyc / .pyo is supported
>by import (if not, then pkgutil should also not support it).

It's an interesting question.  Under Python 2, I'd say yes, it should support
source-less imports, just as Python does.  Under Python 3, with PEP 3147, I'm
not so sure:

$ cat > foo.py
print('hello world')
$ cat > bar.py
import foo
$ ls
bar.py	foo.py
$ python3 bar.py
hello world
$ ls
bar.py	foo.py	__pycache__/
$ ls __pycache__/
foo.cpython-32.pyc
$ rm -f foo.py
$ python3 bar.py
Traceback (most recent call last):
  File "bar.py", line 1, in <module>
    import foo
ImportError: No module named foo

Then:

$ mv __pycache__/foo.cpython-32.pyc foo.pyc
$ ls
bar.py	foo.pyc  __pycache__/
$ python3 bar.py
hello world

So Python 3 source-less imports are still supported, but only when legacy
.pyc/.pyo locations are explicitly used.

My inclination is then -0 on extend_path() supporting .pyc/.pyo but only
*outside* the context of PEP 3147.  I.e. do not use imp.cache_from_source().
History
Date User Action Args
2020-11-16 17:17:26iritkatrielsetversions: + Python 3.8, Python 3.9, Python 3.10, - Python 2.7, Python 3.2, Python 3.3
2011-11-29 15:09:15barrysetmessages: + msg148576
2011-11-29 12:50:45eric.araujosetnosy: + barry, pje, ncoghlan
messages: + msg148556
2011-11-29 08:07:27eric.snowsetnosy: + eric.snow
2011-11-29 06:40:08ezio.melottisetstage: needs patch
2011-06-01 18:17:24eric.araujosetnosy: + eric.araujo
messages: + msg137453
2011-06-01 06:25:13terry.reedysetversions: - Python 2.6, Python 2.5, Python 3.1
2011-03-02 14:59:03Alexandre.Badezsetfiles: + patch.diff

messages: + msg129900
2011-03-02 14:58:37Alexandre.Badezsetfiles: - patch.diff
2011-03-02 14:50:59Alexandre.Badezsetfiles: + patch.diff

messages: + msg129898
keywords: + patch
2011-03-02 14:31:19Alexandre.Badezsettype: behavior
2011-03-02 14:31:05Alexandre.Badezcreate