classification
Title: ModuleFinder.load_module skips incorrect number of bytes in pyc files
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.4, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: Arfrever, bkabrda, brett.cannon, eric.snow, georg.brandl, python-dev, r.david.murray, takluyver
Priority: normal Keywords: 3.3regression, patch

Created on 2014-02-26 09:49 by bkabrda, last changed 2014-03-28 02:11 by takluyver. This issue is now closed.

Files
File name Uploaded Description Edit
fix-bytes-skipped-in-load_module.patch bkabrda, 2014-02-26 09:49 review
Messages (14)
msg212244 - (view) Author: Bohuslav "Slavek" Kabrda (bkabrda) * Date: 2014-02-26 09:49
ModuleFinder.load_module currently only skips 8 bytes before trying to marshal.load the rest of the file, but it should skip 12 since 3.3 (magic, date, file size). I'm attaching a patch with test case.

BTW this was very painful to find out, since I couldn't find any reference to written pyc files - am I searching wrong or is this really not documented anywhere?

(Note, that this was originally reported at https://bugzilla.redhat.com/show_bug.cgi?id=1060338).
msg212256 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-02-26 14:58
It's not documented because the format of .pyc files is considered an internal implementation detail.
msg212259 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-02-26 15:10
Probably want to make sure that modulefinder uses importlib._bootstrap._validate_bytecode_header() to do the parsing.
msg212260 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-02-26 15:14
Since modulefinder is used by freeze tools (notably cx_Freeze, which seems to be the most popular currently), should this be considered a release blocker?
msg212262 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-02-26 15:21
It's been broken since Python 3.3 so this is not a 3.3 regression.
msg212265 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-02-26 15:37
Right.  I'm asking if it should be a release blocker for the next 3.3, too :)

I'm not saying it should be, just raising the question.  It's in my mind because I'm currently using cx_Freeze in a project for a client.  It could have affected me, since I was going to release using 3.3, but as it turns out I have to use 2.7 because of a non-ported dependency that I don't have time to port myself.
msg212268 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-02-26 15:40
Oops, didn't mean to remove the keyword.
msg212274 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-02-26 16:12
Don't know why this is any more special of a bug because it influences cx_freeze compared to any other bug that influences a popular project. I mean I'm not going to stop you from making it a blocker but I'm also not going to rush to fix it myself either.
msg212276 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-02-26 17:07
Well, because of the fact that freeze tools are used to distribute programs on the Windows platform.  But, given that it hasn't been reported before and has been a problem since 3.3, it seems like there is no rush.
msg212447 - (view) Author: Roundup Robot (python-dev) Date: 2014-02-28 15:50
New changeset 432cb56db05d by Brett Cannon in branch '3.3':
Issue #20778: Fix modulefinder to work with bytecode-only modules.
http://hg.python.org/cpython/rev/432cb56db05d

New changeset b6e999c8907c by Brett Cannon in branch 'default':
merge for issue #20778
http://hg.python.org/cpython/rev/b6e999c8907c
msg212448 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-02-28 15:52
Fixed in Python 3.3.6 and 3.4.1.

Bohuslav, could you sign the contributor agreement at http://www.python.org/psf/contrib/contrib-form/ ? While I didn't directly use your patch this time I noticed you are already in Misc/ACKS so it would be helpful if you could sign the document.
msg212449 - (view) Author: Bohuslav "Slavek" Kabrda (bkabrda) * Date: 2014-02-28 15:55
Brett, I signed it just yesterday, it probably wasn't processed yet. Hopefully it will be in few days. Thanks for patching this!
msg212450 - (view) Author: Bohuslav "Slavek" Kabrda (bkabrda) * Date: 2014-02-28 15:56
Whoops, reopened by accident. Closing again.
msg215013 - (view) Author: Thomas Kluyver (takluyver) * Date: 2014-03-28 02:11
For future reference, cx_Freeze ships its own copy of ModuleFinder, so it doesn't depend on the stdlib copy. This issue was fixed there some time around the release of Python 3.3.

I realised recently that this is based on code in the stdlib, and I've been meaning to work out whether cx_Freeze can use any of the stdlib code and lose parts of its own implementation. I guess it's been diverging for quite some time, though.
History
Date User Action Args
2014-03-28 02:11:54takluyversetnosy: + takluyver
messages: + msg215013
2014-02-28 15:56:19bkabrdasetstatus: open -> closed
resolution: fixed
messages: + msg212450
2014-02-28 15:55:27bkabrdasetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg212449
2014-02-28 15:52:49brett.cannonsetstatus: open -> closed
resolution: fixed
messages: + msg212448

stage: resolved
2014-02-28 15:50:43python-devsetnosy: + python-dev
messages: + msg212447
2014-02-27 07:49:47Arfreversetkeywords: + patch
2014-02-27 07:49:38Arfreversetkeywords: + 3.3regression, - patch, 3.2regression
2014-02-26 23:30:46brett.cannonsetassignee: brett.cannon
2014-02-26 17:28:46larrysetnosy: - larry
2014-02-26 17:07:55r.david.murraysetmessages: + msg212276
2014-02-26 16:12:25brett.cannonsetmessages: + msg212274
2014-02-26 15:40:03r.david.murraysetkeywords: + 3.2regression

messages: + msg212268
2014-02-26 15:37:37r.david.murraysetkeywords: - 3.2regression

messages: + msg212265
2014-02-26 15:21:37brett.cannonsetkeywords: + patch
2014-02-26 15:21:12brett.cannonsetkeywords: + 3.2regression, - patch

messages: + msg212262
2014-02-26 15:14:33r.david.murraysetnosy: + r.david.murray
messages: + msg212260
2014-02-26 15:10:39brett.cannonsetmessages: + msg212259
2014-02-26 15:00:08brett.cannonsetpriority: release blocker -> normal
2014-02-26 14:58:35brett.cannonsetpriority: normal -> release blocker
nosy: + larry, georg.brandl
messages: + msg212256

2014-02-26 13:25:00r.david.murraysetnosy: + brett.cannon, eric.snow
2014-02-26 10:40:40Arfreversetnosy: + Arfrever
2014-02-26 09:49:14bkabrdacreate