Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to construct CodeType on Python-3.8.0a4 #81067

Closed
BigStone mannequin opened this issue May 11, 2019 · 16 comments
Closed

Failed to construct CodeType on Python-3.8.0a4 #81067

BigStone mannequin opened this issue May 11, 2019 · 16 comments
Labels
3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error

Comments

@BigStone
Copy link
Mannequin

BigStone mannequin commented May 11, 2019

BPO 36886
Nosy @pfmoore, @vstinner, @tjguk, @zware, @zooba, @Carreau, @pablogsal
PRs
  • bpo-36886: Document changes in code object in What's new section #13255
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-05-12.21:46:04.452>
    created_at = <Date 2019-05-11.10:35:09.684>
    labels = ['interpreter-core', 'invalid', 'type-bug', '3.8']
    title = 'Failed to construct CodeType on Python-3.8.0a4'
    updated_at = <Date 2019-05-24.12:24:17.577>
    user = 'https://bugs.python.org/BigStone'

    bugs.python.org fields:

    activity = <Date 2019-05-24.12:24:17.577>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-05-12.21:46:04.452>
    closer = 'pablogsal'
    components = ['Interpreter Core']
    creation = <Date 2019-05-11.10:35:09.684>
    creator = 'Big Stone'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 36886
    keywords = ['patch']
    message_count = 13.0
    messages = ['342182', '342183', '342222', '342223', '342224', '342225', '342226', '342227', '342229', '342242', '342269', '342272', '343383']
    nosy_count = 9.0
    nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'SilentGhost', 'zach.ware', 'steve.dower', 'mbussonn', 'Big Stone', 'pablogsal']
    pr_nums = ['13255']
    priority = 'normal'
    resolution = 'not a bug'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue36886'
    versions = ['Python 3.8']

    @BigStone
    Copy link
    Mannequin Author

    BigStone mannequin commented May 11, 2019

    remark: I get a similar error from two packages, when experimenting with Python-3.8.0a4 on Windows

    import asyncio
    await asyncio.sleep(0)
    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    C:\WinP\bd38\bu\WPy64-3800a4\python-3.8.0a4.amd64\lib\site-packages\IPython\core\interactiveshell.py in removed_co_newlocals(function)
        139     CO_NEWLOCALS = 0x0002
        140     code = function.__code__
    --> 141     new_code = CodeType(
        142         code.co_argcount,
        143         code.co_kwonlyargcount,
    
    TypeError: an integer is required (got type bytes)
    ```\`
    
    or else:
    
    

    import cloudpickle
    ````

    import cloudpickle
    1 0 1 3 19 b'\x87\x00f\x01d\x01d\x02\x84\x08\x01\x00|\x00\x89\x00d\x00S\x00' (None, <code object <lambda> at 0x000001B36B7D49B0, file "C:\WinP\bd38\bu\WPy64-3800a4\python-3.8.0a4.amd64\lib\site-packages\cloudpickle\cloudpickle.py", line 107>, '_make_cell_set_template_code.<locals>.inner.<locals>.<lambda>') () ('value',) C:\WinP\bd38\bu\WPy64-3800a4\python-3.8.0a4.amd64\lib\site-packages\cloudpickle\cloudpickle.py inner 106 b'\x00\x01\x0c\x01' ('cell',)
    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    <ipython-input-1-316e5ca2d1c8> in <module>
    ----> 1 import cloudpickle
    
    C:\WinP\bd38\bu\WPy64-3800a4\python-3.8.0a4.amd64\lib\site-packages\cloudpickle\__init__.py in <module>
          1 from __future__ import absolute_import
          2 
    ----> 3 from cloudpickle.cloudpickle import *
          4 
          5 __version__ = '1.0.0'
    
    C:\WinP\bd38\bu\WPy64-3800a4\python-3.8.0a4.amd64\lib\site-packages\cloudpickle\cloudpickle.py in <module>
        165 
        166 
    --> 167 _cell_set_template_code = _make_cell_set_template_code()
        168 
        169 
    
    C:\WinP\bd38\bu\WPy64-3800a4\python-3.8.0a4.amd64\lib\site-packages\cloudpickle\cloudpickle.py in _make_cell_set_template_code()
        146             co.co_lnotab,
        147             co.co_cellvars)
    --> 148         return types.CodeType(
        149             co.co_argcount,
        150             co.co_kwonlyargcount,
    
    TypeError: an integer is required (got type bytes)
    ```
    

    @BigStone BigStone mannequin added OS-windows 3.8 only security fixes labels May 11, 2019
    @SilentGhost
    Copy link
    Mannequin

    SilentGhost mannequin commented May 11, 2019

    This is a consequence of PEP-570 that implements positional-only arguments (see bpo-36540). Those projects would have to update their code. Good that there are already open bugs for both projects.

    @SilentGhost SilentGhost mannequin added interpreter-core (Objects, Python, Grammar, and Parser dirs) and removed OS-windows labels May 11, 2019
    @SilentGhost SilentGhost mannequin closed this as completed May 11, 2019
    @SilentGhost SilentGhost mannequin added invalid type-bug An unexpected behavior, bug, or error labels May 11, 2019
    @Carreau
    Copy link
    Mannequin

    Carreau mannequin commented May 11, 2019

    If types is not supposed to be stable, could a warning be added at top of https://docs.python.org/3/library/types.html that it should not be considered stable, and/or to CodeType docs to indicate the signature has changed in 3.8 ?

    If it is supposed to be stable can the posonlyargcount be moved to be the last argument ?

    @pablogsal
    Copy link
    Member

    If it is supposed to be stable can the posonlyargcount be moved to be the last argument ?

    There are other projects that have already added a fix for the new parameter positionally. If we add it to be the last (that still will break the old call) we would break them again.

    @pablogsal
    Copy link
    Member

    and/or to CodeType docs to indicate the signature has changed in 3.8 ?

    The documentation never documented the arguments for the code type (probably on purpose), so not sure if adding a notice there will help much:

    https://docs.python.org/3.8/library/types.html#types.CodeType

    We can maybe add something more clear to the "what's new" section or the changelog.

    @pablogsal pablogsal reopened this May 11, 2019
    @pablogsal
    Copy link
    Member

    As a final note, notice that when keyword-only arguments were implemented, the parameter was not added at the end of the constructor:

    commit 4f72a78
    Author: Guido van Rossum <guido@python.org>
    Date: Fri Oct 27 23:31:49 2006 +0000

    Jiwon Seo's PEP-3102 implementation.
    See SF#1549670.
    The compiler package has not yet been updated.
    

    --- a/Objects/codeobject.c
    +++ b/Objects/codeobject.c
    @@ -41,7 +41,8 @@ intern_strings(PyObject *tuple)

     PyCodeObject *
    -PyCode_New(int argcount, int nlocals, int stacksize, int flags,
    +PyCode_New(int argcount, int kwonlyargcount,
    +          int nlocals, int stacksize, int flags,
               PyObject *code, PyObject *consts, PyObject *names,
               PyObject *varnames, PyObject *freevars, PyObject *cellvars,
               PyObject *filename, PyObject *name, int firstlineno,

    @pablogsal pablogsal changed the title problem with Types on Python-3.8.0a4 Failed to construct CodeType on Python-3.8.0a4 May 11, 2019
    @Carreau
    Copy link
    Mannequin

    Carreau mannequin commented May 11, 2019

    There are other projects that have already added a fix for the new parameter positionally. If we add it to be the last (that still will break the old call) we would break them again.

    Understandable

    The documentation never documented the arguments for the code type (probably on purpose)

    Understandable as well, just thinking that a CYA sentence in types.rst:
    "These types are not supposed to be instantiated outside of CPython internals and constructor signatures can vary between python versions.

    And maybe the docstrings. I rarely open the documentation on python.org and work most of the time by calling help() or ? in IPython, so having some information saying this is non-public in the docstring would make sens to me.

    I'm happy to do it if it is deemed useful.

    @pablogsal
    Copy link
    Member

    Understandable as well, just thinking that a CYA sentence in types.rst:
    "These types are not supposed to be instantiated outside of CPython internals and constructor signatures can vary between python versions.
    And maybe the docstrings. I rarely open the documentation on python.org and work most of the time by calling help() or ? in IPython, so having some information saying this is non-public in the docstring would make sens to me.
    I'm happy to do it if it is deemed useful.

    I am +1 to such a sentence, but I think this is a decision that more core devs should agree on. Taking into account previous history (like the linked commit) and
    the fact that promising stability on such internal pieces will prevent us from making critical optimizations I assume that others will agree, but I am still important
    to know that there is a consensus.

    I would recommend opening such PR if you have some time and let others review it as well.

    @pablogsal
    Copy link
    Member

    After PR 13255 is merged, let's open a new issue for the notice in types.rst to keep the discussion isolated.

    @BigStone
    Copy link
    Mannequin Author

    BigStone mannequin commented May 12, 2019

    I found a third project that is impacted microsoft/python-language-server#1070.

    @pablogsal
    Copy link
    Member

    New changeset 5d23e28 by Pablo Galindo in branch 'master':
    bpo-36886: Document changes in code object in What's new section (GH-13255)
    5d23e28

    @Carreau
    Copy link
    Mannequin

    Carreau mannequin commented May 12, 2019

    Thanks for the update to the what's new Pablo.

    let's open a new issue for the notice in types.rst to keep the discussion isolated.

    See bpo-36896

    @vstinner
    Copy link
    Member

    I created bpo-37032 to add a new CodeType.replace() helper which help projects to be more future-proof (no longer break if CodeType gets yet another parameter).

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @taeyurl
    Copy link

    taeyurl commented May 31, 2022

    Hi, is there a simple fix for the error below?
    Traceback (most recent call last):
    File "zipline/data/bundles/alpaca_api.py", line 12, in
    import zipline.config
    File "c:\users\mpower\zipline-trader\zipline_init_.py", line 30, in
    from .utils.run_algo import run_algorithm
    File "c:\users\mpower\zipline-trader\zipline\utils\run_algo.py", line 24, in
    from zipline.data import bundles
    File "c:\users\mpower\zipline-trader\zipline\data\bundles_init_.py", line 2, in
    from . import quandl # noqa
    File "c:\users\mpower\zipline-trader\zipline\data\bundles\quandl.py", line 16, in
    from . import core as bundles
    File "c:\users\mpower\zipline-trader\zipline\data\bundles\core.py", line 13, in
    from ..adjustments import SQLiteAdjustmentReader, SQLiteAdjustmentWriter
    File "c:\users\mpower\zipline-trader\zipline\data\adjustments.py", line 29, in
    from ._adjustments import load_adjustments_from_sqlite
    File "zipline\data_adjustments.pyx", line 35, in init zipline.data.adjustments
    from zipline.assets.asset_writer import (
    File "c:\users\mpower\zipline-trader\zipline\assets_init
    .py", line 22, in
    from .assets import (
    File "c:\users\mpower\zipline-trader\zipline\assets\assets.py", line 67, in
    from .asset_writer import (
    File "c:\users\mpower\zipline-trader\zipline\assets\asset_writer.py", line 453, in
    class AssetDBWriter(object):
    File "c:\users\mpower\zipline-trader\zipline\assets\asset_writer.py", line 464, in AssetDBWriter
    def init(self, engine, asset_finder=None):
    File "c:\users\mpower\zipline-trader\zipline\utils\preprocess.py", line 109, in _decorator
    return _build_preprocessed_function(
    File "c:\users\mpower\zipline-trader\zipline\utils\preprocess.py", line 247, in _build_preprocessed_function
    new_func.code = CodeType(*map(getitem(args), _code_argorder))
    TypeError: an integer is required (got type bytes)

    @vstinner
    Copy link
    Member

    That's a closed issue, it's the wrong place to ask questions. Please report the issue to the maintainer of the zipline project.

    @taeyurl
    Copy link

    taeyurl commented May 31, 2022 via email

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants