Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use pickle protocol 4 by default? #67592

Closed
serhiy-storchaka opened this issue Feb 6, 2015 · 16 comments
Closed

Use pickle protocol 4 by default? #67592

serhiy-storchaka opened this issue Feb 6, 2015 · 16 comments
Assignees
@ambv
Labels
3.8 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@serhiy-storchaka
Copy link
Member

BPO 23403
Nosy @pitrou, @vstinner, @avassalotti, @ambv, @serhiy-storchaka, @MojoVampire, @applio
PRs
  • bpo-23403: Bump pickle.DEFAULT_PROTOCOL to 4 #6355
  • bpo-23403: Bump pickle.DEFAULT_PROTOCOL to 4 #6355
  • bpo-23403: [lib2to3] Make grammar pickling faster #6491
    		•
    Dependencies
  • bpo-33209: Repeated Pickler.dump() doesn't reset the state in the C implementation of pickle
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/ambv'
closed_at = <Date 2018-07-24.09:17:14.288>
created_at = <Date 2015-02-06.21:28:18.329>
labels = ['3.8', 'type-feature', 'library']
title = 'Use pickle protocol 4 by default?'
updated_at = <Date 2018-07-24.09:17:14.288>
user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2018-07-24.09:17:14.288>
actor = 'serhiy.storchaka'
assignee = 'lukasz.langa'
closed = True
closed_date = <Date 2018-07-24.09:17:14.288>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2015-02-06.21:28:18.329>
creator = 'serhiy.storchaka'
dependencies = ['33209']
files = []
hgrepos = []
issue_num = 23403
keywords = ['patch', 'patch']
message_count = 15.0
messages = ['235498', '235499', '235501', '235502', '235504', '235507', '251858', '275440', '314841', '314842', '314927', '315075', '315080', '315117', '322282']
nosy_count = 9.0
nosy_names = ['pitrou', 'vstinner', 'alexandre.vassalotti', 'Arfrever', 'lukasz.langa', 'sbt', 'serhiy.storchaka', 'josh.r', 'davin']
pr_nums = ['6355', '6355', '6491']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23403'
versions = ['Python 3.8']
    @serhiy-storchaka
    Copy link
    Member Author

    Why not use pickle protocol 4 by default? It allows to pickle some objects which are not pickleable with lower protocols and is more efficient.

    @serhiy-storchaka serhiy-storchaka added stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Feb 6, 2015
    @pitrou
    Copy link
    Member

    pitrou commented Feb 6, 2015

    Because pickles will not be compatible with Python < 3.4 anymore. People who want maximum efficiency without compatibility can pass the right protocol manually.

    @serhiy-storchaka
    Copy link
    Member Author

    Multiprocessing uses default protocol and there is no simple (without hacking the stdlib) way to pass the right protocol manually.

    @pitrou
    Copy link
    Member

    pitrou commented Feb 6, 2015

    Multiprocessing uses default protocol

    AFAIK, this is so you can interact with processes using another version of Python.

    @MojoVampire
    Copy link
    Mannequin

    MojoVampire mannequin commented Feb 6, 2015

    multiprocessing spawns the other processes itself from the same executable used to launch the main process. It's not subprocess. How would a different version of Python get involved?

    @pitrou
    Copy link
    Member

    pitrou commented Feb 7, 2015

    It is possible to have independent processes communicate together, although that's not the most widely-used feature. See:
    https://docs.python.org/3/library/multiprocessing.html#multiprocessing-listeners-clients
    Also:
    https://docs.python.org/3/library/multiprocessing.html#managers

    @vstinner
    Copy link
    Member

    Can't we "negociate" the protocol automatically?

    @applio
    Copy link
    Member

    applio commented Sep 9, 2016

    Closing in deference to the enhancement described in bpo-28053.

    @applio applio closed this as completed Sep 9, 2016
    @ambv
    Copy link
    Contributor

    ambv commented Apr 2, 2018

    Now that Python 3.3 is dead and 3.4 is soon to follow, it's safe to bump DEFAULT_PROTOCOL to 4.

    @ambv ambv reopened this Apr 2, 2018
    @ambv ambv self-assigned this Apr 2, 2018
    @ambv ambv added the 3.8 only security fixes label Apr 2, 2018
    @serhiy-storchaka
    Copy link
    Member Author

    Should we bump the pickle protocol for shelve?

    @ambv
    Copy link
    Contributor

    ambv commented Apr 4, 2018

    New changeset c51d8c9 by Łukasz Langa in branch 'master':
    bpo-23403: Bump pickle.DEFAULT_PROTOCOL to 4 (bpo-6355)
    c51d8c9

    @pitrou
    Copy link
    Member

    pitrou commented Apr 7, 2018

    Is there something left to be done here?

    @serhiy-storchaka
    Copy link
    Member Author

    Shelve still uses protocol 3 by default. Should it be bumped too?

    @pitrou
    Copy link
    Member

    pitrou commented Apr 9, 2018

    Shelve still uses protocol 3 by default. Should it be bumped too?

    That sounds reasonable. Perhaps open a separate issue for it?

    @serhiy-storchaka
    Copy link
    Member Author

    Opened bpo-34204 for shelve.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @gpshead
    Copy link
    Member

    gpshead commented May 14, 2022

    It is possible to have independent processes communicate together, although that's not the most widely-used feature. See:
    https://docs.python.org/3/library/multiprocessing.html#multiprocessing-listeners-clients
    Also:
    https://docs.python.org/3/library/multiprocessing.html#managers

    Wow. I really wish we did not support this. multiprocessing was not intended to be a general purpose RPC mechanism between uncoordinated processes.

    @gpshead gpshead mentioned this issue May 14, 2022
    Open
    3 tasks
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @ambv ambv

    Labels
    3.8 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    6 participants
    @ambv @gpshead @applio @vstinner @pitrou @serhiy-storchaka