classification
Title: compileall.compile_dir(..., ddir="") omits the intermediate package paths when prepending the prefix
Type: behavior Stage: commit review
Components: Library (Lib) Versions: Python 3.9, Python 3.8, Python 3.7, Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: gregory.p.smith
Priority: normal Keywords: patch

Created on 2020-02-27 05:18 by gregory.p.smith, last changed 2020-03-01 19:07 by gregory.p.smith. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 18676 merged gregory.p.smith, 2020-02-27 08:47
PR 18718 merged gregory.p.smith, 2020-02-29 21:59
PR 18725 merged gregory.p.smith, 2020-03-01 18:46
Messages (11)
msg362767 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-02-27 05:18
Easiest to demonstrate as such:

```shell
#!/bin/bash

mkdir bug
touch bug/__init__.py
mkdir bug/foo
touch bug/foo/__init__.py
touch bug/foo/bar.py

python3 -m compileall -d "<prefix>" bug
python2 -m compileall -d "<prefix>" bug

echo "prefix embedded in PY3 pyc code object for lib.foo.bar:"
strings bug/foo/__pycache__/bar.cpython-3*.pyc | grep prefix
echo "prefix embedded in PY2 pyc code object for lib.foo.bar:"
strings bug/foo/bar.pyc | grep prefix
```

Run that script and you'll see:
Listing 'bug'...
Compiling 'bug/__init__.py'...
Listing 'bug/foo'...
Compiling 'bug/foo/__init__.py'...
Compiling 'bug/foo/bar.py'...
Listing bug ...
Compiling bug/__init__.py ...
Listing bug/__pycache__ ...
Listing bug/foo ...
Compiling bug/foo/__init__.py ...
Listing bug/foo/__pycache__ ...
Compiling bug/foo/bar.py ...
prefix embedded in PY3 pyc code object for lib.foo.bar:
<prefix>/bar.py
prefix embedded in PY2 pyc code object for lib.foo.bar:
<prefix>/foo/bar.pyt


Notice that the Python 3 pyc file contains a code.co_filename of "<prefix>/bar.py" instead of the correct value (that Python 2 inserts) of "<prefix>/foo/bar.py".
msg362768 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-02-27 05:20
This has probably been happening for many 3 releases.  I noticed it on 3.6 when investigating a problem with wrong paths in tracebacks coming from code which we used compileall.compile_dir(src_root, quiet=1, ddir="<prefix">) on.

I'm guessing ddir= (-d on the cli) isn't widely used?
msg362782 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-02-27 07:30
Ironically 3.9a4 gains some compileall enhancements that could be used as an awkward workaround from https://bugs.python.org/issue38112:

python 2.7 compile_dir(d, ddir="<foo>") behavior can be had with
python 3.9a4 compile_dir(d, prependdir="<foo>", stripdir=d)

I still intend to fix the actual ddir behavior as it is wrong.
msg362785 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-02-27 08:02
The regression was caused by the parallelization enhancement implementation from https://bugs.python.org/issue16104
msg362787 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-02-27 08:02
meaning this regression was introduced in 3.5.
msg362788 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-02-27 08:04
belated postmortem: if there had been tests against the filename encoded in the pycs living in subdirs as generated by compile_dir, this regression would not have happened.
msg362956 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-02-29 01:28
New changeset 02673352b5db6ca4d3dc804965facbedfe66425d by Gregory P. Smith in branch 'master':
bpo-39769: Fix compileall ddir for subpkgs. (GH-18676)
https://github.com/python/cpython/commit/02673352b5db6ca4d3dc804965facbedfe66425d
msg362961 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-02-29 02:13
keeping this open while i investigate if fixing 3.8 and 3.7 is feasible.  if nothing else i'll try to add a note to the docs about the issue in 3.5-3.8 with a code sample suggested workaround.
msg363077 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-03-01 18:43
New changeset ce720d3e0674d6ac6f1b950c20a89be4cfde7853 by Gregory P. Smith in branch '3.8':
bpo-39769: Fix compileall ddir for subpkgs. (GH-18676) (GH-18718)
https://github.com/python/cpython/commit/ce720d3e0674d6ac6f1b950c20a89be4cfde7853
msg363078 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-03-01 19:06
New changeset 7c64726ced3d6b5d04537386d6a9ca6d179c3be4 by Gregory P. Smith in branch '3.7':
[3.7] bpo-39769: Fix compileall ddir for subpkgs. (GH-18676) (GH-18718) (GH-18725)
https://github.com/python/cpython/commit/7c64726ced3d6b5d04537386d6a9ca6d179c3be4
msg363079 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-03-01 19:07
if anyone needs this on their older 3.6 or 3.5 trees, the 3.7/3.8 patch is a trivial backport.
History
Date User Action Args
2020-03-01 19:07:52gregory.p.smithsetstatus: open -> closed

messages: + msg363079
stage: patch review -> commit review
2020-03-01 19:06:58gregory.p.smithsetmessages: + msg363078
2020-03-01 18:46:50gregory.p.smithsetpull_requests: + pull_request18082
2020-03-01 18:43:07gregory.p.smithsetmessages: + msg363077
2020-02-29 21:59:05gregory.p.smithsetstage: commit review -> patch review
pull_requests: + pull_request18075
2020-02-29 02:13:07gregory.p.smithsetresolution: fixed
messages: + msg362961
stage: patch review -> commit review
2020-02-29 01:28:40gregory.p.smithsetmessages: + msg362956
2020-02-27 08:47:19gregory.p.smithsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request18036
2020-02-27 08:04:38gregory.p.smithsetmessages: + msg362788
2020-02-27 08:02:58gregory.p.smithsetmessages: + msg362787
versions: + Python 3.5
2020-02-27 08:02:00gregory.p.smithsetmessages: + msg362785
2020-02-27 07:30:37gregory.p.smithsetmessages: + msg362782
2020-02-27 05:20:15gregory.p.smithsettype: behavior
messages: + msg362768
components: + Library (Lib)
stage: needs patch
2020-02-27 05:18:05gregory.p.smithcreate