Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reloading tokenize breaks tokenize.open() #67803

Closed
takluyver mannequin opened this issue Mar 9, 2015 · 11 comments
Closed

Reloading tokenize breaks tokenize.open() #67803

takluyver mannequin opened this issue Mar 9, 2015 · 11 comments
Assignees
@serhiy-storchaka
Labels
stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@takluyver
Copy link
Mannequin

takluyver mannequin commented Mar 9, 2015

BPO 23615
Nosy @brettcannon, @ncoghlan, @vstinner, @ericsnowcurrently, @takluyver, @serhiy-storchaka
Files
  • tokenize-reloadable.patch
  • tokenize-reloadable-B.patch
  • tokenize-reloadable-B2.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2015-03-11.15:35:00.110>
created_at = <Date 2015-03-09.01:21:08.683>
labels = ['type-bug', 'library']
title = 'Reloading tokenize breaks tokenize.open()'
updated_at = <Date 2015-03-11.21:54:20.069>
user = 'https://github.com/takluyver'

    bugs.python.org fields:

    activity = <Date 2015-03-11.21:54:20.069>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2015-03-11.15:35:00.110>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2015-03-09.01:21:08.683>
creator = 'takluyver'
dependencies = []
files = ['38423', '38425', '38426']
hgrepos = []
issue_num = 23615
keywords = ['patch']
message_count = 11.0
messages = ['237588', '237782', '237784', '237787', '237788', '237790', '237798', '237870', '237876', '237904', '237906']
nosy_count = 7.0
nosy_names = ['brett.cannon', 'ncoghlan', 'vstinner', 'python-dev', 'eric.snow', 'takluyver', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue23615'
versions = ['Python 3.4', 'Python 3.5']
    @takluyver
    Copy link
    Mannequin Author

    takluyver mannequin commented Mar 9, 2015

    Issue bpo-22599 changed tokenize.open() from using builtins.open() to having a module-level reference to _builtin_open, stored by doing _builtin_open = open. However, on reloading the module, _builtin_open is pointed to tokenize.open from the last execution of the code, breaking tokenize.open():

    >>> import tokenize
>>> tokenize.open
<function open at 0x7f15b660fc80>
>>> tokenize._builtin_open
<built-in function open>
>>> import imp; imp.reload(tokenize)
<module 'tokenize' from '/home/takluyver/miniconda3/envs/py34/lib/python3.4/tokenize.py'>
>>> tokenize.open
<function open at 0x7f15b660fbf8>
>>> tokenize._builtin_open
<function open at 0x7f15b660fc80>
>>> tokenize.open('foo.py')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/takluyver/miniconda3/envs/py34/lib/python3.4/tokenize.py", line 438, in open
    buffer = _builtin_open(filename, 'rb')
TypeError: open() takes 1 positional argument but 2 were given

    The actual case where I'm seeing this error is nose logging a failure in a test suite, so it's not clear what is reloading tokenize, but it appears that something is. This just started failing when our Windows test system updated to Python 3.4.3.

    @takluyver takluyver mannequin added the stdlib Python modules in the Lib dir label Mar 9, 2015
    @takluyver
    Copy link
    Mannequin Author

    takluyver mannequin commented Mar 10, 2015

    Patch attached to fix this.

    @serhiy-storchaka
    Copy link
    Member

    The same issue exists in Lib/bz2.py, Lib/tarfile.py, Tools/freeze/bkfile.py.

    May be write a code as

    from builtins import open as _builtin_open

    ?

    @serhiy-storchaka serhiy-storchaka added the type-bug An unexpected behavior, bug, or error label Mar 10, 2015
    @takluyver
    Copy link
    Mannequin Author

    takluyver mannequin commented Mar 10, 2015

    -B.patch as Serhiy suggests, for tokenize only for the time being.

    @serhiy-storchaka
    Copy link
    Member

    LGTM. Do you want to write a patch for other cases Thomas?

    @takluyver
    Copy link
    Mannequin Author

    takluyver mannequin commented Mar 10, 2015

    Fixed the other three cases you pointed out (-B2.patch).

    @serhiy-storchaka
    Copy link
    Member

    LGTM.

    @brettcannon
    Copy link
    Member

    LGTM as well. You want to commit, Serhiy? If not assign to me and I will get to it on Friday.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Mar 11, 2015

    New changeset 383ba3699084 by Serhiy Storchaka in branch '3.4':
    Issue bpo-23615: Modules bz2, tarfile and tokenize now can be reloaded with
    https://hg.python.org/cpython/rev/383ba3699084

    New changeset 6e736a57a482 by Serhiy Storchaka in branch 'default':
    Issue bpo-23615: Modules bz2, tarfile and tokenize now can be reloaded with
    https://hg.python.org/cpython/rev/6e736a57a482

    New changeset 36bd5add9732 by Serhiy Storchaka in branch '2.7':
    Issue bpo-23615: Module tarfile is now can be reloaded with imp.reload().
    https://hg.python.org/cpython/rev/36bd5add9732

    @vstinner
    Copy link
    Member

    Is there a method to detect other reload bugs?

    @serhiy-storchaka
    Copy link
    Member

    Only grep.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @serhiy-storchaka serhiy-storchaka

    Labels
    stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @brettcannon @vstinner @serhiy-storchaka