classification
Title: Failed to construct CodeType on Python-3.8.0a4
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.8
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Big Stone, SilentGhost, mbussonn, pablogsal, paul.moore, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords: patch

Created on 2019-05-11 10:35 by Big Stone, last changed 2019-05-24 12:24 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 13255 merged pablogsal, 2019-05-11 22:48
Messages (13)
msg342182 - (view) Author: Big Stone (Big Stone) Date: 2019-05-11 10:35
remark: I get a similar error from two packages, when experimenting with Python-3.8.0a4 on Windows

````
import asyncio
await asyncio.sleep(0)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
C:\WinP\bd38\bu\WPy64-3800a4\python-3.8.0a4.amd64\lib\site-packages\IPython\core\interactiveshell.py in removed_co_newlocals(function)
    139     CO_NEWLOCALS = 0x0002
    140     code = function.__code__
--> 141     new_code = CodeType(
    142         code.co_argcount,
    143         code.co_kwonlyargcount,

TypeError: an integer is required (got type bytes)
````

or else:

````
import cloudpickle
````

````
import cloudpickle
1 0 1 3 19 b'\x87\x00f\x01d\x01d\x02\x84\x08\x01\x00|\x00\x89\x00d\x00S\x00' (None, <code object <lambda> at 0x000001B36B7D49B0, file "C:\WinP\bd38\bu\WPy64-3800a4\python-3.8.0a4.amd64\lib\site-packages\cloudpickle\cloudpickle.py", line 107>, '_make_cell_set_template_code.<locals>.inner.<locals>.<lambda>') () ('value',) C:\WinP\bd38\bu\WPy64-3800a4\python-3.8.0a4.amd64\lib\site-packages\cloudpickle\cloudpickle.py inner 106 b'\x00\x01\x0c\x01' ('cell',)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-316e5ca2d1c8> in <module>
----> 1 import cloudpickle

C:\WinP\bd38\bu\WPy64-3800a4\python-3.8.0a4.amd64\lib\site-packages\cloudpickle\__init__.py in <module>
      1 from __future__ import absolute_import
      2 
----> 3 from cloudpickle.cloudpickle import *
      4 
      5 __version__ = '1.0.0'

C:\WinP\bd38\bu\WPy64-3800a4\python-3.8.0a4.amd64\lib\site-packages\cloudpickle\cloudpickle.py in <module>
    165 
    166 
--> 167 _cell_set_template_code = _make_cell_set_template_code()
    168 
    169 

C:\WinP\bd38\bu\WPy64-3800a4\python-3.8.0a4.amd64\lib\site-packages\cloudpickle\cloudpickle.py in _make_cell_set_template_code()
    146             co.co_lnotab,
    147             co.co_cellvars)
--> 148         return types.CodeType(
    149             co.co_argcount,
    150             co.co_kwonlyargcount,

TypeError: an integer is required (got type bytes)
msg342183 - (view) Author: SilentGhost (SilentGhost) * (Python triager) Date: 2019-05-11 10:46
This is a consequence of PEP 570 that implements positional-only arguments (see issue 36540). Those projects would have to update their code. Good that there are already open bugs for both projects.
msg342222 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2019-05-11 21:11
If types is not supposed to be stable, could a warning be added at top of https://docs.python.org/3/library/types.html that it should not be considered stable, and/or to CodeType docs to indicate the signature has changed in 3.8 ?

If it is supposed to be stable can the `posonlyargcount` be moved to   be the last argument ?
msg342223 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-05-11 22:28
> If it is supposed to be stable can the `posonlyargcount` be moved to   be the last argument ?

There are other projects that have already added a fix for the new parameter positionally. If we add it to be the last (that still will break the old call) we would break them again.
msg342224 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-05-11 22:31
> and/or to CodeType docs to indicate the signature has changed in 3.8 ?

The documentation never documented the arguments for the code type (probably on purpose), so not sure if adding a notice there will help much:

https://docs.python.org/3.8/library/types.html#types.CodeType

We can maybe add something more clear to the "what's new" section or the changelog.
msg342225 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-05-11 22:40
As a final note, notice that when keyword-only arguments were implemented, the parameter was not added at the end of the constructor:

commit 4f72a78684bbfcdc43ceeabb240ceee54706c4b0
Author: Guido van Rossum <guido@python.org>
Date:   Fri Oct 27 23:31:49 2006 +0000

    Jiwon Seo's PEP 3102 implementation.
    See SF#1549670.
    The compiler package has not yet been updated.


--- a/Objects/codeobject.c
+++ b/Objects/codeobject.c
@@ -41,7 +41,8 @@ intern_strings(PyObject *tuple)


 PyCodeObject *
-PyCode_New(int argcount, int nlocals, int stacksize, int flags,
+PyCode_New(int argcount, int kwonlyargcount,
+          int nlocals, int stacksize, int flags,
           PyObject *code, PyObject *consts, PyObject *names,
           PyObject *varnames, PyObject *freevars, PyObject *cellvars,
           PyObject *filename, PyObject *name, int firstlineno,
msg342226 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2019-05-11 22:49
> There are other projects that have already added a fix for the new parameter positionally. If we add it to be the last (that still will break the old call) we would break them again.

Understandable


> The documentation never documented the arguments for the code type (probably on purpose)

Understandable as well, just thinking that a CYA sentence in types.rst: 
"These types are not supposed to be instantiated outside of CPython internals and constructor signatures can vary between python versions.

And maybe the docstrings. I rarely open the documentation on python.org and work most of the time by calling `help()` or `?` in IPython, so having some information saying this is non-public in the docstring would make sens to me. 

I'm happy to do it if it is deemed useful.
msg342227 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-05-11 22:58
>Understandable as well, just thinking that a CYA sentence in types.rst: 
>"These types are not supposed to be instantiated outside of CPython internals and constructor signatures can vary between python versions.
>And maybe the docstrings. I rarely open the documentation on python.org and work most of the time by calling `help()` or `?` in IPython, so having some information saying this is non-public in the docstring would make sens to me. 
>I'm happy to do it if it is deemed useful.

I am +1 to such a sentence, but I think this is a decision that more core devs should agree on. Taking into account previous history (like the linked commit) and
the fact that promising stability on such internal pieces will prevent us from making critical optimizations I assume that others will agree, but I am still important
to know that there is a consensus.

I would recommend opening such PR if you have some time and let others review it as well.
msg342229 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-05-11 23:05
After PR 13255 is merged, let's open a new issue for the notice in types.rst to keep the discussion isolated.
msg342242 - (view) Author: Big Stone (Big Stone) Date: 2019-05-12 06:07
I found a third project that is impacted https://github.com/microsoft/python-language-server/issues/1070.
msg342269 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-05-12 21:45
New changeset 5d23e282af69d404a3430bb95aefe371112817b3 by Pablo Galindo in branch 'master':
bpo-36886: Document changes in code object in What's new section (GH-13255)
https://github.com/python/cpython/commit/5d23e282af69d404a3430bb95aefe371112817b3
msg342272 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2019-05-12 22:04
Thanks for the update to the what's new Pablo. 

> let's open a new issue for the notice in types.rst to keep the discussion isolated.

See bpo-36896
msg343383 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-24 12:24
I created bpo-37032 to add a new CodeType.replace() helper which help projects to be more future-proof (no longer break if CodeType gets yet another parameter).
History
Date User Action Args
2019-05-24 12:24:17vstinnersetnosy: + vstinner
messages: + msg343383
2019-05-12 22:04:46mbussonnsetmessages: + msg342272
2019-05-12 21:46:04pablogsalsetstatus: open -> closed
stage: patch review -> resolved
2019-05-12 21:45:55pablogsalsetmessages: + msg342269
2019-05-12 06:07:46Big Stonesetmessages: + msg342242
2019-05-11 23:05:38pablogsalsetmessages: + msg342229
2019-05-11 22:58:20pablogsalsetmessages: + msg342227
2019-05-11 22:49:56mbussonnsetmessages: + msg342226
2019-05-11 22:48:39pablogsalsetkeywords: + patch
stage: resolved -> patch review
pull_requests: + pull_request13167
2019-05-11 22:41:02pablogsalsettitle: problem with Types on Python-3.8.0a4 -> Failed to construct CodeType on Python-3.8.0a4
2019-05-11 22:40:06pablogsalsetmessages: + msg342225
2019-05-11 22:32:04pablogsalsetstatus: closed -> open
2019-05-11 22:31:56pablogsalsetmessages: + msg342224
2019-05-11 22:28:36pablogsalsetmessages: + msg342223
2019-05-11 21:11:45mbussonnsetnosy: + mbussonn
messages: + msg342222
2019-05-11 10:46:08SilentGhostsetstatus: open -> closed

type: behavior
components: + Interpreter Core, - Windows

nosy: + pablogsal, SilentGhost
messages: + msg342183
resolution: not a bug
stage: resolved
2019-05-11 10:35:09Big Stonecreate