classification
Title: Add CodeType.replace() method
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: blueyed, mbussonn, pablogsal, vstinner
Priority: normal Keywords: patch

Created on 2019-05-24 09:25 by vstinner, last changed 2019-06-06 14:15 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 13542 merged vstinner, 2019-05-24 10:41
Messages (15)
msg343360 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-24 09:25
Each type types.CodeType constructor changes, a lot of applications break. Python 3.8 added a new parameter which broke for example the Genshi project, just to name one.

I propose to add a new CodeType.replace() method, similar to datetime.datetime.replace and namedtuple._replace() for example:

$ python3
Python 3.7.3 (default, Mar 27 2019, 13:41:07) 

>>> import datetime
>>> d=datetime.datetime.now()
>>> d2=d.replace(year=2010)
>>> d, d2
(datetime.datetime(2019, 5, 24, 11, 25, 3, 839966), datetime.datetime(2010, 5, 24, 11, 25, 3, 839966))

>>> import collections
>>> Point = collections.namedtuple("Point", "x y")
>>> p=Point(1, 2)
>>> p2=p._replace(y=3)
>>> p, p2
(Point(x=1, y=2), Point(x=1, y=3))
msg343370 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-24 10:56
> Python 3.8 added a new parameter which broke for example the Genshi project

Pablo fixed it in Genshi:
https://github.com/edgewall/genshi/pull/19
msg343371 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-24 10:59
See also bpo-37034: "Argument Clinic omits name of keyword-only parameter on _PyArg_BadArgument() call".
msg343373 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-24 11:11
Fix in Hypothesis:
https://github.com/HypothesisWorks/hypothesis/commit/8f47297fa2e19c426a42b06bb5f8bf1406b8f0f3#diff-a097452b8ffe8323641ccccac335fbf9
msg343374 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-24 11:14
ipython fix:
https://github.com/ipython/ipython/commit/248128dfaabb33e922b1e36a298fd7ec0c730069
msg343377 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-24 11:39
It's not like Python 3.8 is the first type in Python history that code constructor changed.

The PyCode_New() function slowly growed from 3 parameters in 1990 to 16 paramters in 2019 :-)

History of PyCode_New().


New "posonlyargcount" parameter:

commit 8c77b8cb9188165a123f2512026e3629bf03dc9b
Author: Pablo Galindo <Pablogsal@gmail.com>
Date:   Mon Apr 29 13:36:57 2019 +0100

    bpo-36540: PEP 570 -- Implementation (GH-12701)


New "kwonlyargcount" parameter, bpo-1549670:

commit 4f72a78684bbfcdc43ceeabb240ceee54706c4b0
Author: Guido van Rossum <guido@python.org>
Date:   Fri Oct 27 23:31:49 2006 +0000

    Jiwon Seo's PEP 3102 implementation.
    See SF#1549670.
    The compiler package has not yet been updated.


New "freevars" and "cellvars" parameters, PEP 227:

commit 64949cb753f206c0ca1d83f55d07afd3c179b81a
Author: Jeremy Hylton <jeremy@alum.mit.edu>
Date:   Thu Jan 25 20:06:59 2001 +0000

    PEP 227 implementation
    ...
    Include/compile.h
        Add co_freevars and co_cellvars slots to code objects.
        Update PyCode_New() to take freevars and cellvars as arguments


New "firstlineno" and "lnotab" parameters:

commit da4eb5c3b55df9b7d6957ddd0458313102f0a92e
Author: Guido van Rossum <guido@python.org>
Date:   Fri Jan 24 03:43:35 1997 +0000

    Instead of emitting SET_LINENO instructions, generate a line number
    table which is incorporated in the code object.  This way, the runtime
    overhead to keep track of line numbers is only incurred when an
    exception has to be reported.


New "stacksize" parameter:

commit 8b993a98db507cc3a75067af4c865ffc8afbada1
Author: Guido van Rossum <guido@python.org>
Date:   Fri Jan 17 21:04:03 1997 +0000

    Add co_stacksize field to codeobject structure, and stacksize argument
    to PyCode_New() argument list.  Move MAXBLOCKS constant to conpile.h.
    
    Added accurate calculation of the actual stack size needed by the
    generated code.
    
    Also commented out all fprintf statements (except for a new one to
    diagnose stack underflow, and one in #ifdef'ed out code), and added
    some new TO DO suggestions (now that the stacksize is taken of the TO
    DO list).


New "argcount", "nlocals", "flags" and "varnames" parameters:

commit 681d79aaf397850778608f35585d091fa7fe370a
Author: Guido van Rossum <guido@python.org>
Date:   Tue Jul 18 14:51:37 1995 +0000

    keyword arguments and faster calls


New "name" parameter:

commit 9bfef44d97d1ae24e03717e3d59024b44626a9de
Author: Guido van Rossum <guido@python.org>
Date:   Mon Mar 29 10:43:31 1993 +0000

    ...
    * Added function name to code object.  Print it for code and function
      objects.  THIS MAKES THE .PYC FILE FORMAT INCOMPATIBLE (the version
      number has changed accordingly)
    ...


New "filename" parameter:

commit 3f5da24ea304e674a9abbdcffc4d671e32aa70f1
Author: Guido van Rossum <guido@python.org>
Date:   Thu Dec 20 15:06:42 1990 +0000

    "Compiling" version


The very first version of the function was:

static codeobject *
newcodeobject(code, consts, names)
  object *code;
  object *consts;
  object *names;

From this commit:

commit 10dc2e8097e7a431367e72f46ddba91be93aa159
Author: Guido van Rossum <guido@python.org>
Date:   Sun Nov 18 17:27:39 1990 +0000

    Initial revision
msg343378 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-24 11:40
> It's not like Python 3.8 is the first type in Python history that code constructor changed.

Oops, typo: It's not like Python 3.8 is the first *time* in Python history that the code constructor changed.
msg343381 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-24 11:46
Fix in Cloud Pickle:
https://github.com/cloudpipe/cloudpickle/commit/b9dc17fc5f723ffbfc665295fafdd076907c0a93
msg343384 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-24 12:25
See bpo-36886 "Failed to construct CodeType on Python-3.8.0a4" for other failures caused by the addition of the "posonlyargcount" parameter to code constructor.
msg343434 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-24 21:57
New changeset a9f05d69ccbf3c75cdd604c25094282697789a62 by Victor Stinner in branch 'master':
bpo-37032: Add CodeType.replace() method (GH-13542)
https://github.com/python/cpython/commit/a9f05d69ccbf3c75cdd604c25094282697789a62
msg343435 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-24 22:02
Once Python 3.8 beta1 will be released, it would be interesting to patch all projects that I had to be fixed to handle the new "posonlyargcount" to use the new "replace()" method is available.

Maybe even remove the fix for Python 3.8 alpha 4 and only use use replace() on Python 3.8 and newer.
msg343589 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-05-27 00:17
Is there anything more on this issue?
msg343610 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-27 07:45
Pablo: "Is there anything more on this issue?"

I proposed "Once Python 3.8 beta1 will be released, it would be interesting to patch all projects that I had to be fixed to handle the new "posonlyargcount" to use the new "replace()" method is available."

But that only concerns third party project,s no Python directly, so I close the issue :-)
msg344816 - (view) Author: daniel hahler (blueyed) * Date: 2019-06-06 14:07
Thanks for this great improvement!

Was about to suggest having something more futureproof when noticing this with pdbpp, found this, and submitted PRs to use it for ipython and hypothesis also.
msg344817 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-06 14:15
> submitted PRs to use it for ipython and hypothesis also

Cool! Thank you for doing that ;-)
History
Date User Action Args
2019-06-06 14:15:03vstinnersetmessages: + msg344817
2019-06-06 14:07:13blueyedsetnosy: + blueyed
messages: + msg344816
2019-05-27 07:45:56vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg343610

stage: patch review -> resolved
2019-05-27 00:17:15pablogsalsetmessages: + msg343589
2019-05-26 20:10:20mbussonnsetnosy: + mbussonn
2019-05-24 22:02:59vstinnersetmessages: + msg343435
2019-05-24 21:57:26vstinnersetmessages: + msg343434
2019-05-24 12:25:04vstinnersetmessages: + msg343384
2019-05-24 11:46:05vstinnersetmessages: + msg343381
2019-05-24 11:40:05vstinnersetmessages: + msg343378
2019-05-24 11:39:29vstinnersetmessages: + msg343377
2019-05-24 11:14:59vstinnersetmessages: + msg343374
2019-05-24 11:11:28vstinnersetmessages: + msg343373
2019-05-24 10:59:14vstinnersetmessages: + msg343371
2019-05-24 10:56:41vstinnersetmessages: + msg343370
2019-05-24 10:41:42vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request13456
2019-05-24 09:25:55vstinnercreate