classification
Title: Reloading tokenize breaks tokenize.open()
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.5, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: brett.cannon, eric.snow, haypo, ncoghlan, python-dev, serhiy.storchaka, takluyver
Priority: normal Keywords: patch

Created on 2015-03-09 01:21 by takluyver, last changed 2015-03-11 21:54 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
tokenize-reloadable.patch takluyver, 2015-03-10 17:11 review
tokenize-reloadable-B.patch takluyver, 2015-03-10 17:51 review
tokenize-reloadable-B2.patch takluyver, 2015-03-10 18:14 review
Messages (11)
msg237588 - (view) Author: Thomas Kluyver (takluyver) * Date: 2015-03-09 01:21
Issue #22599 changed tokenize.open() from using builtins.open() to having a module-level reference to _builtin_open, stored by doing _builtin_open = open. However, on reloading the module, _builtin_open is pointed to tokenize.open from the last execution of the code, breaking tokenize.open():

>>> import tokenize
>>> tokenize.open
<function open at 0x7f15b660fc80>
>>> tokenize._builtin_open
<built-in function open>
>>> import imp; imp.reload(tokenize)
<module 'tokenize' from '/home/takluyver/miniconda3/envs/py34/lib/python3.4/tokenize.py'>
>>> tokenize.open
<function open at 0x7f15b660fbf8>
>>> tokenize._builtin_open
<function open at 0x7f15b660fc80>
>>> tokenize.open('foo.py')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/takluyver/miniconda3/envs/py34/lib/python3.4/tokenize.py", line 438, in open
    buffer = _builtin_open(filename, 'rb')
TypeError: open() takes 1 positional argument but 2 were given

The actual case where I'm seeing this error is nose logging a failure in a test suite, so it's not clear what is reloading tokenize, but it appears that something is. This just started failing when our Windows test system updated to Python 3.4.3.
msg237782 - (view) Author: Thomas Kluyver (takluyver) * Date: 2015-03-10 17:11
Patch attached to fix this.
msg237784 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-10 17:41
The same issue exists in Lib/bz2.py, Lib/tarfile.py, Tools/freeze/bkfile.py.

May be write a code as

from builtins import open as _builtin_open

?
msg237787 - (view) Author: Thomas Kluyver (takluyver) * Date: 2015-03-10 17:51
-B.patch as Serhiy suggests, for tokenize only for the time being.
msg237788 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-10 17:57
LGTM. Do you want to write a patch for other cases Thomas?
msg237790 - (view) Author: Thomas Kluyver (takluyver) * Date: 2015-03-10 18:14
Fixed the other three cases you pointed out (-B2.patch).
msg237798 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-10 19:08
LGTM.
msg237870 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-03-11 14:54
LGTM as well. You want to commit, Serhiy? If not assign to me and I will get to it on Friday.
msg237876 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-11 15:32
New changeset 383ba3699084 by Serhiy Storchaka in branch '3.4':
Issue #23615: Modules bz2, tarfile and tokenize now can be reloaded with
https://hg.python.org/cpython/rev/383ba3699084

New changeset 6e736a57a482 by Serhiy Storchaka in branch 'default':
Issue #23615: Modules bz2, tarfile and tokenize now can be reloaded with
https://hg.python.org/cpython/rev/6e736a57a482

New changeset 36bd5add9732 by Serhiy Storchaka in branch '2.7':
Issue #23615: Module tarfile is now can be reloaded with imp.reload().
https://hg.python.org/cpython/rev/36bd5add9732
msg237904 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-11 21:38
Is there a method to detect other reload bugs?
msg237906 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-11 21:54
Only grep.
History
Date User Action Args
2015-03-26 15:01:10r.david.murraylinkissue23784 superseder
2015-03-11 21:54:20serhiy.storchakasetmessages: + msg237906
2015-03-11 21:38:26hayposetmessages: + msg237904
2015-03-11 15:35:00serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: commit review -> resolved
2015-03-11 15:32:55python-devsetnosy: + python-dev
messages: + msg237876
2015-03-11 14:54:44brett.cannonsetassignee: serhiy.storchaka
messages: + msg237870
2015-03-10 19:08:33serhiy.storchakasetmessages: + msg237798
stage: commit review
2015-03-10 18:14:33takluyversetfiles: + tokenize-reloadable-B2.patch

messages: + msg237790
2015-03-10 17:57:31serhiy.storchakasetmessages: + msg237788
2015-03-10 17:51:57takluyversetfiles: + tokenize-reloadable-B.patch

messages: + msg237787
2015-03-10 17:41:59serhiy.storchakasetnosy: + eric.snow, brett.cannon, serhiy.storchaka, ncoghlan
type: behavior
messages: + msg237784
2015-03-10 17:11:33takluyversetfiles: + tokenize-reloadable.patch
keywords: + patch
messages: + msg237782
2015-03-09 01:21:08takluyvercreate