classification
Title: assert statements missed when loaded by zipimporter
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.7, Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, berker.peksag, brett.cannon, gregory.p.smith, jaraco, larry, ned.deily, python-dev, steve.dower, twouters, xiang.zhang
Priority: release blocker Keywords: patch

Created on 2016-09-13 16:19 by jaraco, last changed 2017-03-31 16:36 by dstufft. This issue is now closed.

Files
File name Uploaded Description Edit
issue28131.diff berker.peksag, 2016-09-14 04:37 review
Pull Requests
URL Status Linked Edit
PR 552 closed dstufft, 2017-03-31 16:36
Messages (10)
msg276296 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2016-09-13 16:19
Grabbing the recently released Python 3.6.0b1, I tried running one of my test suites, but found that some assertions were failing to assert when the package was loaded as a zip file (such as with pytest-runner installed dependencies). I distilled the issue to this:

$ cat > mod.py
def test(val):
        assert(val)
        print(val)
$ zip mod.zip mod.py
updating: mod.py (deflated 20%)
$ rm mod.py
$ python
Python 3.6.0b1 (v3.6.0b1:5b0ca4ed5e2f, Sep 12 2016, 09:24:46) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('mod.zip')
>>> import mod
>>> mod.test(False)
False
>>> mod.__loader__
<zipimporter object "mod.zip">
>>> sys.flags.optimize
0

I would have expected the call to mod.test to have raised an AssertionError, and on Python 3.5 it does.

I searched the what's new and didn't see anything advertising this change, so I suspect it's an unintentional regression.

I'm including Brett for his familiarity with importlib, but welcome re-assignment.

If I can do more to help, let me know.
msg276311 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-09-13 18:02
I've added Greg and Thomas in case they have any ideas as they have looked at zipimport more recently than I have.
msg276359 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2016-09-13 22:29
This shouldn't be happening and makes no sense.  It looks like the assert statement was removed at import code compilation time given the pdb trace with it from a zip file vs with it outside of a zip file:


>>> pdb.run('mod.test(False)')
> <string>(1)<module>()
(Pdb) n
False
--Return--
> <string>(1)<module>()->None
(Pdb) n
>>> pdb.run('mod.test(False)')
> <string>(1)<module>()->None
(Pdb) s
--Call--
> oss/cpython/build/mod.zip/mod.py(1)test()
-> def test(val):
(Pdb) s
> oss/cpython/build/mod.zip/mod.py(3)test()
-> print(val)
(Pdb) s
False
--Return--
> oss/cpython/build/mod.zip/mod.py(3)test()->None
-> print(val)
(Pdb) s
--Return--
> <string>(1)<module>()->None
(Pdb) s
> oss/cpython/default/Lib/bdb.py(435)run()
-> self.quitting = True
(Pdb) s
>>> 

vs no zip file:

>>> pdb.run('mod.test(False)')
> <string>(1)<module>()
(Pdb) s
--Call--
> oss/cpython/build/mod.py(1)test()
-> def test(val):
(Pdb) s
> oss/cpython/build/mod.py(2)test()
-> assert(val)
(Pdb) s
AssertionError
> oss/cpython/build/mod.py(2)test()
-> assert(val)
(Pdb) s
--Return--
> oss/cpython/build/mod.py(2)test()->None
-> assert(val)
(Pdb) s
AssertionError
> <string>(1)<module>()
(Pdb) s
--Return--
> <string>(1)<module>()->None
(Pdb) s
AssertionError
> oss/cpython/default/Lib/bdb.py(431)run()
-> exec(cmd, globals, locals)
(Pdb) s
> oss/cpython/default/Lib/bdb.py(432)run()
-> except BdbQuit:
(Pdb) s
> oss/cpython/default/Lib/bdb.py(435)run()
-> self.quitting = True
(Pdb) s
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "oss/cpython/default/Lib/pdb.py", line 1568, in run
    Pdb().run(statement, globals, locals)
  File "oss/cpython/default/Lib/bdb.py", line 431, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "oss/cpython/build/mod.py", line 2, in test
    assert(val)
AssertionError
msg276360 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2016-09-13 22:31
from the zip:

>>> dis.dis(mod.test)
  3           0 LOAD_GLOBAL              0 (print)
              2 LOAD_FAST                0 (val)
              4 CALL_FUNCTION            1
              6 POP_TOP
              8 LOAD_CONST               0 (None)
             10 RETURN_VALUE

from the filesystem:
>>> dis.dis(mod.test)
  2           0 LOAD_FAST                0 (val)
              2 POP_JUMP_IF_TRUE         8
              4 LOAD_GLOBAL              0 (AssertionError)
              6 RAISE_VARARGS            1

  3     >>    8 LOAD_GLOBAL              1 (print)
             10 LOAD_FAST                0 (val)
             12 CALL_FUNCTION            1
             14 POP_TOP
             16 LOAD_CONST               0 (None)
             18 RETURN_VALUE
msg276369 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2016-09-14 03:00
This is introduced in 663a62bcf9c9. The compile optimize flag is opened in the change.
msg276371 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-09-14 03:56
Bah, that's meant to be a -1 in that change, not 1. I must have typo'd when moving from the default branch to put it in 3.5.

Anyone's welcome to fix it, or I'll get it when I have time for CPython work.
msg276372 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-09-14 04:37
Here is a patch with a test case.
msg276376 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2016-09-14 04:58
lgtm
msg276377 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-14 05:10
New changeset 7bec326972f5 by Berker Peksag in branch '3.5':
Issue #28131: Fix a regression in zipimport's compile_source()
https://hg.python.org/cpython/rev/7bec326972f5

New changeset 7a6c0c4e6072 by Berker Peksag in branch '3.6':
Issue #28131: Merge from 3.5
https://hg.python.org/cpython/rev/7a6c0c4e6072

New changeset 873760b02024 by Berker Peksag in branch 'default':
Issue #28131: Merge from 3.6
https://hg.python.org/cpython/rev/873760b02024
msg276378 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-09-14 05:14
Thanks for the report and for the reviews!
History
Date User Action Args
2017-03-31 16:36:13dstufftsetpull_requests: + pull_request887
2016-09-14 05:14:50berker.peksagsetstatus: open -> closed
resolution: fixed
messages: + msg276378

stage: patch review -> resolved
2016-09-14 05:10:32python-devsetnosy: + python-dev
messages: + msg276377
2016-09-14 04:58:58benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg276376
2016-09-14 04:37:56berker.peksagsetfiles: + issue28131.diff

type: behavior

keywords: + patch
nosy: + berker.peksag
messages: + msg276372
stage: patch review
2016-09-14 03:56:34steve.dowersetnosy: + larry

versions: + Python 3.5
2016-09-14 03:56:15steve.dowersetmessages: + msg276371
2016-09-14 03:00:08xiang.zhangsetnosy: + xiang.zhang, steve.dower
messages: + msg276369
2016-09-13 22:31:07gregory.p.smithsetmessages: + msg276360
2016-09-13 22:29:49gregory.p.smithsetmessages: + msg276359
2016-09-13 18:02:47brett.cannonsetnosy: + twouters, gregory.p.smith
messages: + msg276311
2016-09-13 16:19:15jaracocreate