Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ModuleFinder.load_module skips incorrect number of bytes in pyc files #64977

Closed
bkabrda mannequin opened this issue Feb 26, 2014 · 14 comments
Closed

ModuleFinder.load_module skips incorrect number of bytes in pyc files #64977

bkabrda mannequin opened this issue Feb 26, 2014 · 14 comments
Assignees
Labels
stdlib Python modules in the Lib dir

Comments

@bkabrda
Copy link
Mannequin

bkabrda mannequin commented Feb 26, 2014

BPO 20778
Nosy @brettcannon, @birkenfeld, @bitdancer, @ericsnowcurrently, @takluyver
Files
  • fix-bytes-skipped-in-load_module.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/brettcannon'
    closed_at = <Date 2014-02-28.15:56:19.335>
    created_at = <Date 2014-02-26.09:49:14.733>
    labels = ['library']
    title = 'ModuleFinder.load_module skips incorrect number of bytes in pyc files'
    updated_at = <Date 2014-03-28.02:11:54.737>
    user = 'https://bugs.python.org/bkabrda'

    bugs.python.org fields:

    activity = <Date 2014-03-28.02:11:54.737>
    actor = 'takluyver'
    assignee = 'brett.cannon'
    closed = True
    closed_date = <Date 2014-02-28.15:56:19.335>
    closer = 'bkabrda'
    components = ['Library (Lib)']
    creation = <Date 2014-02-26.09:49:14.733>
    creator = 'bkabrda'
    dependencies = []
    files = ['34226']
    hgrepos = []
    issue_num = 20778
    keywords = ['patch', '3.3regression']
    message_count = 14.0
    messages = ['212244', '212256', '212259', '212260', '212262', '212265', '212268', '212274', '212276', '212447', '212448', '212449', '212450', '215013']
    nosy_count = 8.0
    nosy_names = ['brett.cannon', 'georg.brandl', 'Arfrever', 'r.david.murray', 'python-dev', 'eric.snow', 'takluyver', 'bkabrda']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue20778'
    versions = ['Python 3.3', 'Python 3.4']

    @bkabrda
    Copy link
    Mannequin Author

    bkabrda mannequin commented Feb 26, 2014

    ModuleFinder.load_module currently only skips 8 bytes before trying to marshal.load the rest of the file, but it should skip 12 since 3.3 (magic, date, file size). I'm attaching a patch with test case.

    BTW this was very painful to find out, since I couldn't find any reference to written pyc files - am I searching wrong or is this really not documented anywhere?

    (Note, that this was originally reported at https://bugzilla.redhat.com/show_bug.cgi?id=1060338).

    @bkabrda bkabrda mannequin added the stdlib Python modules in the Lib dir label Feb 26, 2014
    @brettcannon
    Copy link
    Member

    It's not documented because the format of .pyc files is considered an internal implementation detail.

    @brettcannon
    Copy link
    Member

    Probably want to make sure that modulefinder uses importlib._bootstrap._validate_bytecode_header() to do the parsing.

    @bitdancer
    Copy link
    Member

    Since modulefinder is used by freeze tools (notably cx_Freeze, which seems to be the most popular currently), should this be considered a release blocker?

    @brettcannon
    Copy link
    Member

    It's been broken since Python 3.3 so this is not a 3.3 regression.

    @bitdancer
    Copy link
    Member

    Right. I'm asking if it should be a release blocker for the next 3.3, too :)

    I'm not saying it should be, just raising the question. It's in my mind because I'm currently using cx_Freeze in a project for a client. It could have affected me, since I was going to release using 3.3, but as it turns out I have to use 2.7 because of a non-ported dependency that I don't have time to port myself.

    @bitdancer
    Copy link
    Member

    Oops, didn't mean to remove the keyword.

    @brettcannon
    Copy link
    Member

    Don't know why this is any more special of a bug because it influences cx_freeze compared to any other bug that influences a popular project. I mean I'm not going to stop you from making it a blocker but I'm also not going to rush to fix it myself either.

    @bitdancer
    Copy link
    Member

    Well, because of the fact that freeze tools are used to distribute programs on the Windows platform. But, given that it hasn't been reported before and has been a problem since 3.3, it seems like there is no rush.

    @brettcannon brettcannon self-assigned this Feb 26, 2014
    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Feb 28, 2014

    New changeset 432cb56db05d by Brett Cannon in branch '3.3':
    Issue bpo-20778: Fix modulefinder to work with bytecode-only modules.
    http://hg.python.org/cpython/rev/432cb56db05d

    New changeset b6e999c8907c by Brett Cannon in branch 'default':
    merge for issue bpo-20778
    http://hg.python.org/cpython/rev/b6e999c8907c

    @brettcannon
    Copy link
    Member

    Fixed in Python 3.3.6 and 3.4.1.

    Bohuslav, could you sign the contributor agreement at http://www.python.org/psf/contrib/contrib-form/ ? While I didn't directly use your patch this time I noticed you are already in Misc/ACKS so it would be helpful if you could sign the document.

    @bkabrda
    Copy link
    Mannequin Author

    bkabrda mannequin commented Feb 28, 2014

    Brett, I signed it just yesterday, it probably wasn't processed yet. Hopefully it will be in few days. Thanks for patching this!

    @bkabrda bkabrda mannequin reopened this Feb 28, 2014
    @bkabrda
    Copy link
    Mannequin Author

    bkabrda mannequin commented Feb 28, 2014

    Whoops, reopened by accident. Closing again.

    @bkabrda bkabrda mannequin closed this as completed Feb 28, 2014
    @takluyver
    Copy link
    Mannequin

    takluyver mannequin commented Mar 28, 2014

    For future reference, cx_Freeze ships its own copy of ModuleFinder, so it doesn't depend on the stdlib copy. This issue was fixed there some time around the release of Python 3.3.

    I realised recently that this is based on code in the stdlib, and I've been meaning to work out whether cx_Freeze can use any of the stdlib code and lose parts of its own implementation. I guess it's been diverging for quite some time, though.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    stdlib Python modules in the Lib dir
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants