Navigation Menu

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CodeType.replace() method #81213

Closed
vstinner opened this issue May 24, 2019 · 17 comments
Closed

Add CodeType.replace() method #81213

vstinner opened this issue May 24, 2019 · 17 comments
Labels
3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs)

Comments

@vstinner
Copy link
Member

BPO 37032
Nosy @vstinner, @blueyed, @Carreau, @pablogsal
PRs
  • bpo-37032: Add CodeType.replace() method #13542
  • bpo-37032: Document CodeType.replace (new in python 3.8) #17776
  • [3.8] bpo-37032: Document CodeType.replace (GH-17776) #17779
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-05-27.07:45:56.864>
    created_at = <Date 2019-05-24.09:25:55.826>
    labels = ['interpreter-core', '3.8']
    title = 'Add CodeType.replace() method'
    updated_at = <Date 2020-01-01.06:12:20.408>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2020-01-01.06:12:20.408>
    actor = 'miss-islington'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-05-27.07:45:56.864>
    closer = 'vstinner'
    components = ['Interpreter Core']
    creation = <Date 2019-05-24.09:25:55.826>
    creator = 'vstinner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 37032
    keywords = ['patch']
    message_count = 15.0
    messages = ['343360', '343370', '343371', '343373', '343374', '343377', '343378', '343381', '343384', '343434', '343435', '343589', '343610', '344816', '344817']
    nosy_count = 4.0
    nosy_names = ['vstinner', 'blueyed', 'mbussonn', 'pablogsal']
    pr_nums = ['13542', '17776', '17779']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue37032'
    versions = ['Python 3.8']

    @vstinner
    Copy link
    Member Author

    Each type types.CodeType constructor changes, a lot of applications break. Python 3.8 added a new parameter which broke for example the Genshi project, just to name one.

    I propose to add a new CodeType.replace() method, similar to datetime.datetime.replace and namedtuple._replace() for example:

    $ python3
    Python 3.7.3 (default, Mar 27 2019, 13:41:07) 
    >>> import datetime
    >>> d=datetime.datetime.now()
    >>> d2=d.replace(year=2010)
    >>> d, d2
    (datetime.datetime(2019, 5, 24, 11, 25, 3, 839966), datetime.datetime(2010, 5, 24, 11, 25, 3, 839966))
    
    >>> import collections
    >>> Point = collections.namedtuple("Point", "x y")
    >>> p=Point(1, 2)
    >>> p2=p._replace(y=3)
    >>> p, p2
    (Point(x=1, y=2), Point(x=1, y=3))

    @vstinner vstinner added 3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) labels May 24, 2019
    @vstinner
    Copy link
    Member Author

    Python 3.8 added a new parameter which broke for example the Genshi project

    Pablo fixed it in Genshi:
    edgewall/genshi#19

    @vstinner
    Copy link
    Member Author

    See also bpo-37034: "Argument Clinic omits name of keyword-only parameter on _PyArg_BadArgument() call".

    @vstinner
    Copy link
    Member Author

    @vstinner
    Copy link
    Member Author

    ipython fix:
    ipython/ipython@248128d

    @vstinner
    Copy link
    Member Author

    It's not like Python 3.8 is the first type in Python history that code constructor changed.

    The PyCode_New() function slowly growed from 3 parameters in 1990 to 16 paramters in 2019 :-)

    History of PyCode_New().

    New "posonlyargcount" parameter:

    commit 8c77b8c
    Author: Pablo Galindo <Pablogsal@gmail.com>
    Date: Mon Apr 29 13:36:57 2019 +0100

    bpo-36540: PEP-570 -- Implementation (GH-12701)
    

    New "kwonlyargcount" parameter, bpo-1549670:

    commit 4f72a78
    Author: Guido van Rossum <guido@python.org>
    Date: Fri Oct 27 23:31:49 2006 +0000

    Jiwon Seo's PEP-3102 implementation.
    See SF#1549670.
    The compiler package has not yet been updated.
    

    New "freevars" and "cellvars" parameters, PEP-227:

    commit 64949cb
    Author: Jeremy Hylton <jeremy@alum.mit.edu>
    Date: Thu Jan 25 20:06:59 2001 +0000

    PEP-227 implementation
    ...
    [Include/compile.h](https://github.com/python/cpython/blob/main/Include/compile.h)
        Add co_freevars and co_cellvars slots to code objects.
        Update PyCode_New() to take freevars and cellvars as arguments
    

    New "firstlineno" and "lnotab" parameters:

    commit da4eb5c
    Author: Guido van Rossum <guido@python.org>
    Date: Fri Jan 24 03:43:35 1997 +0000

    Instead of emitting SET_LINENO instructions, generate a line number
    table which is incorporated in the code object.  This way, the runtime
    overhead to keep track of line numbers is only incurred when an
    exception has to be reported.
    

    New "stacksize" parameter:

    commit 8b993a9
    Author: Guido van Rossum <guido@python.org>
    Date: Fri Jan 17 21:04:03 1997 +0000

    Add co_stacksize field to codeobject structure, and stacksize argument
    to PyCode_New() argument list.  Move MAXBLOCKS constant to conpile.h.
    
    Added accurate calculation of the actual stack size needed by the
    generated code.
    
    Also commented out all fprintf statements (except for a new one to
    diagnose stack underflow, and one in #ifdef'ed out code), and added
    some new TO DO suggestions (now that the stacksize is taken of the TO
    DO list).
    

    New "argcount", "nlocals", "flags" and "varnames" parameters:

    commit 681d79a
    Author: Guido van Rossum <guido@python.org>
    Date: Tue Jul 18 14:51:37 1995 +0000

    keyword arguments and faster calls
    

    New "name" parameter:

    commit 9bfef44
    Author: Guido van Rossum <guido@python.org>
    Date: Mon Mar 29 10:43:31 1993 +0000

    ...
    * Added function name to code object.  Print it for code and function
      objects.  THIS MAKES THE .PYC FILE FORMAT INCOMPATIBLE (the version
      number has changed accordingly)
    ...
    

    New "filename" parameter:

    commit 3f5da24
    Author: Guido van Rossum <guido@python.org>
    Date: Thu Dec 20 15:06:42 1990 +0000

    "Compiling" version
    

    The very first version of the function was:

    static codeobject *
    newcodeobject(code, consts, names)
    object *code;
    object *consts;
    object *names;

    From this commit:

    commit 10dc2e8
    Author: Guido van Rossum <guido@python.org>
    Date: Sun Nov 18 17:27:39 1990 +0000

    Initial revision
    

    @vstinner
    Copy link
    Member Author

    It's not like Python 3.8 is the first type in Python history that code constructor changed.

    Oops, typo: It's not like Python 3.8 is the first *time* in Python history that the code constructor changed.

    @vstinner
    Copy link
    Member Author

    Fix in Cloud Pickle:
    cloudpipe/cloudpickle@b9dc17f

    @vstinner
    Copy link
    Member Author

    See bpo-36886 "Failed to construct CodeType on Python-3.8.0a4" for other failures caused by the addition of the "posonlyargcount" parameter to code constructor.

    @vstinner
    Copy link
    Member Author

    New changeset a9f05d6 by Victor Stinner in branch 'master':
    bpo-37032: Add CodeType.replace() method (GH-13542)
    a9f05d6

    @vstinner
    Copy link
    Member Author

    Once Python 3.8 beta1 will be released, it would be interesting to patch all projects that I had to be fixed to handle the new "posonlyargcount" to use the new "replace()" method is available.

    Maybe even remove the fix for Python 3.8 alpha 4 and only use use replace() on Python 3.8 and newer.

    @pablogsal
    Copy link
    Member

    Is there anything more on this issue?

    @vstinner
    Copy link
    Member Author

    Pablo: "Is there anything more on this issue?"

    I proposed "Once Python 3.8 beta1 will be released, it would be interesting to patch all projects that I had to be fixed to handle the new "posonlyargcount" to use the new "replace()" method is available."

    But that only concerns third party project,s no Python directly, so I close the issue :-)

    @blueyed
    Copy link
    Mannequin

    blueyed mannequin commented Jun 6, 2019

    Thanks for this great improvement!

    Was about to suggest having something more futureproof when noticing this with pdbpp, found this, and submitted PRs to use it for ipython and hypothesis also.

    @vstinner
    Copy link
    Member Author

    vstinner commented Jun 6, 2019

    submitted PRs to use it for ipython and hypothesis also

    Cool! Thank you for doing that ;-)

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @feisuzhu
    Copy link

    feisuzhu commented Nov 9, 2022

    Found this great improvement years after my frustration, many thanks for this work!

    @vstinner
    Copy link
    Member Author

    vstinner commented Nov 9, 2022

    Found this great improvement years after my frustration, many thanks for this work!

    I'm glad that it helped you ;-)

    feisuzhu added a commit to feisuzhu/thbattle that referenced this issue Nov 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs)
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants