Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

compileall.compile_dir(..., ddir="<prefix>") omits the intermediate package paths when prepending the prefix #83950

Closed
gpshead opened this issue Feb 27, 2020 · 11 comments
Closed

compileall.compile_dir(..., ddir="<prefix>") omits the intermediate package paths when prepending the prefix #83950

gpshead opened this issue Feb 27, 2020 · 11 comments
Labels
3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@gpshead
Copy link
Member

gpshead commented Feb 27, 2020

BPO 39769
Nosy @gpshead
PRs
  • bpo-39769: Fix compileall ddir for subpkgs. #18676
  • [3.8] bpo-39769: Fix compileall ddir for subpkgs. (GH-18676) #18718
  • [3.7] bpo-39769: Fix compileall ddir for subpkgs. (GH-18676) (GH-18718) #18725
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2020-03-01.19:07:52.955>
created_at = <Date 2020-02-27.05:18:05.962>
labels = ['3.7', '3.8', 'type-bug', 'library', '3.9']
title = 'compileall.compile_dir(..., ddir="<prefix>") omits the intermediate package paths when prepending the prefix'
updated_at = <Date 2020-03-01.19:07:52.954>
user = 'https://github.com/gpshead'

    bugs.python.org fields:

    activity = <Date 2020-03-01.19:07:52.954>
actor = 'gregory.p.smith'
assignee = 'none'
closed = True
closed_date = <Date 2020-03-01.19:07:52.955>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2020-02-27.05:18:05.962>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 39769
keywords = ['patch']
message_count = 11.0
messages = ['362767', '362768', '362782', '362785', '362787', '362788', '362956', '362961', '363077', '363078', '363079']
nosy_count = 1.0
nosy_names = ['gregory.p.smith']
pr_nums = ['18676', '18718', '18725']
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39769'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9']
    @gpshead
    Copy link
    Member Author

    gpshead commented Feb 27, 2020

    Easiest to demonstrate as such:

    #!/bin/bash

mkdir bug
touch bug/__init__.py
mkdir bug/foo
touch bug/foo/__init__.py
touch bug/foo/bar.py

python3 -m compileall -d "<prefix>" bug
python2 -m compileall -d "<prefix>" bug

echo "prefix embedded in PY3 pyc code object for lib.foo.bar:"
strings bug/foo/__pycache__/bar.cpython-3*.pyc | grep prefix
echo "prefix embedded in PY2 pyc code object for lib.foo.bar:"
strings bug/foo/bar.pyc | grep prefix

    Run that script and you'll see:
    Listing 'bug'...
    Compiling 'bug/init.py'...
    Listing 'bug/foo'...
    Compiling 'bug/foo/init.py'...
    Compiling 'bug/foo/bar.py'...
    Listing bug ...
    Compiling bug/init.py ...
    Listing bug/pycache ...
    Listing bug/foo ...
    Compiling bug/foo/init.py ...
    Listing bug/foo/pycache ...
    Compiling bug/foo/bar.py ...
    prefix embedded in PY3 pyc code object for lib.foo.bar:
    <prefix>/bar.py
    prefix embedded in PY2 pyc code object for lib.foo.bar:
    <prefix>/foo/bar.pyt

    Notice that the Python 3 pyc file contains a code.co_filename of "<prefix>/bar.py" instead of the correct value (that Python 2 inserts) of "<prefix>/foo/bar.py".

    @gpshead gpshead added 3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes labels Feb 27, 2020
    @gpshead
    Copy link
    Member Author

    gpshead commented Feb 27, 2020

    This has probably been happening for many 3 releases. I noticed it on 3.6 when investigating a problem with wrong paths in tracebacks coming from code which we used compileall.compile_dir(src_root, quiet=1, ddir="<prefix">) on.

    I'm guessing ddir= (-d on the cli) isn't widely used?

    @gpshead gpshead added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Feb 27, 2020
    @gpshead
    Copy link
    Member Author

    gpshead commented Feb 27, 2020

    Ironically 3.9a4 gains some compileall enhancements that could be used as an awkward workaround from https://bugs.python.org/issue38112:

    python 2.7 compile_dir(d, ddir="<foo>") behavior can be had with
    python 3.9a4 compile_dir(d, prependdir="<foo>", stripdir=d)

    I still intend to fix the actual ddir behavior as it is wrong.

    @gpshead
    Copy link
    Member Author

    gpshead commented Feb 27, 2020

    The regression was caused by the parallelization enhancement implementation from https://bugs.python.org/issue16104

    @gpshead
    Copy link
    Member Author

    gpshead commented Feb 27, 2020

    meaning this regression was introduced in 3.5.

    @gpshead
    Copy link
    Member Author

    gpshead commented Feb 27, 2020

    belated postmortem: if there had been tests against the filename encoded in the pycs living in subdirs as generated by compile_dir, this regression would not have happened.

    @gpshead
    Copy link
    Member Author

    gpshead commented Feb 29, 2020

    New changeset 0267335 by Gregory P. Smith in branch 'master':
    bpo-39769: Fix compileall ddir for subpkgs. (GH-18676)
    0267335

    @gpshead
    Copy link
    Member Author

    gpshead commented Feb 29, 2020

    keeping this open while i investigate if fixing 3.8 and 3.7 is feasible. if nothing else i'll try to add a note to the docs about the issue in 3.5-3.8 with a code sample suggested workaround.

    @gpshead
    Copy link
    Member Author

    gpshead commented Mar 1, 2020

    New changeset ce720d3 by Gregory P. Smith in branch '3.8':
    bpo-39769: Fix compileall ddir for subpkgs. (GH-18676) (GH-18718)
    ce720d3

    @gpshead
    Copy link
    Member Author

    gpshead commented Mar 1, 2020

    New changeset 7c64726 by Gregory P. Smith in branch '3.7':
    [3.7] bpo-39769: Fix compileall ddir for subpkgs. (GH-18676) (GH-18718) (GH-18725)
    7c64726

    @gpshead
    Copy link
    Member Author

    gpshead commented Mar 1, 2020

    if anyone needs this on their older 3.6 or 3.5 trees, the 3.7/3.8 patch is a trivial backport.

    @gpshead gpshead closed this as completed Mar 1, 2020
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @hetmankp hetmankp mentioned this issue Oct 4, 2022
    Open
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @gpshead