classification
Title: Deprecate lib2to3 (and 2to3) for future removal
Type: enhancement Stage: patch review
Components: 2to3 (2.x to 3.x conversion tool) Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: BTaskaya, Peter Ludemann, carljm, corona10, eric.snow, gregory.p.smith, gvanrossum, hroncok, vstinner
Priority: normal Keywords: patch

Created on 2020-04-22 04:40 by gregory.p.smith, last changed 2020-05-07 23:27 by vstinner.

Pull Requests
URL Status Linked Edit
PR 19645 closed gregory.p.smith, 2020-04-22 05:01
PR 19663 merged carljm, 2020-04-22 20:50
PR 19898 merged hroncok, 2020-05-04 10:02
Messages (20)
msg366973 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-04-22 04:40
Based on the PEP 617 acceptance thread on python-dev, lib2to3 is eventually going to run into trouble parsing modern syntax a few releases from now.

It would be better off maintained outside of the standard library.  It gets used by a lot of things and is generally useful, but would make a lot more sense as a PyPI project than as something only quasi-maintained within the stdlib (it only gained the ability to parse a couple modern syntax features in via bugfix contributions to the stdlib the past month or two...  meaning a lot of versions of it out there cannot)

Black has already forked it.

goal:  PendingDeprecationWarning and documentation as such in 3.9.  Move to DeprecationWarning in 3.10 or 3.11 and remove it by ~3.12.  Subject to our existing deprecation process guidelines.
msg367005 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-22 14:26
I am in favor of this. We could promote LibCST, which is based on Parso, which uses a forked version of pgen2 (the parser in lib2to3). I believe one of these could switch to a fork of pegen as its parser, so it will be able to handle new PEG based syntax in 3.10+.

Removal by 3.12 might be feasible.
msg367031 - (view) Author: Carl Meyer (carljm) * Date: 2020-04-22 17:31
I volunteered in the python-dev thread to write a patch to the docs clarifying future status of lib2to3; happy to include the PendingDeprecationWarning as well.

Re linking to alternatives, we want to make sure we link to alternatives that are committed to updating to support newer Python versions' syntax. This definitely includes LibCST; I can inquire with the parso maintainer about whether it also includes parso. In future it could also include a third-party-maintained copy of lib2to3, if someone picks that up.
msg367051 - (view) Author: Carl Meyer (carljm) * Date: 2020-04-22 21:15
I opened a PR. It deprecates the lib2to3 library to discourage future use of it for Python3, but not the 2to3 tool. This of course means that the lib2to3 module will in practice stick around in the stdlib as long as 2to3 is still bundled with Python.

It seems like the idea in this issue is to deprecate and remove both. I'm not sure what we typically do to deprecate a command-line utility bundled with Python. Given warnings are silent by default, the deprecation warning for lib2to3 won't be visible to users of 2to3. Should I add something to its `--help` output? Or something more aggressive; an unconditionally-printed warning?
msg367208 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-04-24 18:19
New changeset 503de7149d03bdcc671dcbbb5b64f761bb192b4d by Carl Meyer in branch 'master':
bpo-40360: Deprecate lib2to3 module in light of PEP 617 (GH-19663)
https://github.com/python/cpython/commit/503de7149d03bdcc671dcbbb5b64f761bb192b4d
msg367209 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-04-24 18:21
Okay,the pending deprecation is in.  Keeping open as a reminder to turn that into a real DeprecationWarning in 3.10 after the 3.9 branch is cut.

We'll then want to track reminding us to remove it in 3.12.
msg367230 - (view) Author: Carl Meyer (carljm) * Date: 2020-04-24 21:15
@gregory.p.smith

What do you think about the question I raised above about how to make this deprecation visible to users of the 2to3 CLI tool, assuming the plan is to remove both?
msg367235 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-04-24 22:55
I think what we're doing with the documentation update is fine.  We can add a warning on stderr to the tool in 3.11.  But I don't expect people will be using the tool _from_ the latest CPython 3.x by then.

2to3 is already included with Python 2.7 and the only real use for it is for people who still have code they maintain on 2.7 so they've got a copy already.  There is no value in running a 2to3 shipped with Python 3 vs the latest 2.7.  Meaningful updates to it were already back ported to 2.7 over time as it was intentionally exempt from feature freeze.

We should have sorted out a PyPI home for lib2to3 by 3.11 time and can also create a PyPI package for the 2to3 tool itself at that point.

I _think_ there is support for running 2to3 on sources at package install time from setup.py?  But I don't expect anything actually maintained and widely used to require that by the time this deprecation lands.  If it does, that becomes a plumbing issue within package tools to know that requiring 2to3 at either build or install time adds an implicit tool dependency on the new pypi package to get it.

Maybe I'm just in a good mood about all of this, but none of this seems worrisome.
msg367716 - (view) Author: Peter Ludemann (Peter Ludemann) Date: 2020-04-29 23:51
The documentation change gives two possible successors:

https://libcst.readthedocs.io/ (https://github.com/Instagram/LibCST)
https://parso.readthedocs.io/

And I've also seen this mentioned: https://github.com/pyga/awpa

Is it possible to settle on one of these as the successor to the lib2to3 parser? It would be nice to avoid a 2nd deprecation in the future ...
msg367726 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-30 01:39
It's typically not up to the core devs to pick a winning third party library; we tend to recommend libraries that are already essentially category winners, like requests. In a sense pointing to LibCST *and* parso is redundant because LibCST builds on parso. Comparing stars on GitHub:
- LibCST: 423
- parso: 296
- awpa: 10
msg367730 - (view) Author: Carl Meyer (carljm) * Date: 2020-04-30 03:14
Right, although I think it still makes sense to link both LibCST and parso since they provide different levels of abstraction that would be suitable for different types of tools (e.g. I would rather write an auto-formatter on top of parso, because LibCST's careful parsing and assignment of whitespace would mostly just get in the way, but I'd rather write any kind of refactoring tooling on top of LibCST.)

Another tool that escaped my mind when writing the PR that should probably be linked also is Baron/RedBaron (https://github.com/PyCQA/redbaron); 457 stars makes it slightly more popular than LibCST (but it's also been around a lot longer.)
msg367743 - (view) Author: Miro Hrončok (hroncok) * Date: 2020-04-30 07:27
Coul you please add a what's new entry for this change?
msg367744 - (view) Author: Miro Hrončok (hroncok) * Date: 2020-04-30 07:35
I don't understand why there is a PendingDeprecationWarning and not a DeprecationWarning.


See https://discuss.python.org/t/pendingdeprecationwarning-is-really-useful/1038/4 and issue36404
msg367766 - (view) Author: Carl Meyer (carljm) * Date: 2020-04-30 17:59
> Coul you please add a what's new entry for this change?

The committed change already included an entry in NEWS. Is a "What's New" entry something different?

> I don't understand why there is a PendingDeprecationWarning and not a DeprecationWarning.

Purely because I was following gps' recommendation in the first comment on this issue. Getting rid of PendingDeprecationWarning seems like an orthogonal decision; if it happens, this can trivially be upgraded to DeprecationWarning as part of a removal sweep.
msg367767 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-30 18:31
A "What's New" entry would go into Doc/whatsnew/3.9.rst and is much more visible to users looking for exciting bits in the new release (the NEWS file is very large, see e.g. https://docs.python.org/3/whatsnew/changelog.html#changelog.

The What's New doc typically has a section collecting all the deprecations, e.g. https://docs.python.org/3/whatsnew/3.8.html#deprecated.
msg367770 - (view) Author: Miro Hrončok (hroncok) * Date: 2020-04-30 18:44
> Getting rid of PendingDeprecationWarning seems like an orthogonal decision; if it happens, this can trivially be upgraded to DeprecationWarning as part of a removal sweep.

My thought was that the decision was already made to do so. Hence adding new PendingDeprecationWarnings goes against that decision.

But maybe I misunderstand and that decision was not made.
msg367771 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-30 19:08
IIRC PendingDeprecationError does not mean that the decision hasn't been made yet. It just means it's less urgent for folks to worry about. I believe we tend to change PendingDeprecationError to DeprecationError in the last release before something is removed.
msg367884 - (view) Author: Miro Hrončok (hroncok) * Date: 2020-05-01 20:28
Thanks for the explanation.

I plan to send a PR to add this to the What's new in 3.9 page early next week. Anyone, feel free to beat me to it.
msg368077 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-05-04 19:02
New changeset 18f1c60a1625d341a905c7e07367c32c08f222df by Miro Hrončok in branch 'master':
bpo-40360: Add a What's New entry for lib2to3 pending deprecation (GH-19898)
https://github.com/python/cpython/commit/18f1c60a1625d341a905c7e07367c32c08f222df
msg368388 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-07 23:27
FYI the autopep8 project uses lib2to3.
History
Date User Action Args
2020-05-07 23:27:54vstinnersetnosy: + vstinner
messages: + msg368388
2020-05-04 19:02:08gregory.p.smithsetmessages: + msg368077
2020-05-04 10:02:37hroncoksetpull_requests: + pull_request19209
2020-05-01 20:28:08hroncoksetmessages: + msg367884
2020-04-30 19:08:37gvanrossumsetmessages: + msg367771
2020-04-30 18:44:30hroncoksetmessages: + msg367770
2020-04-30 18:31:34gvanrossumsetmessages: + msg367767
2020-04-30 17:59:53carljmsetmessages: + msg367766
2020-04-30 07:35:50hroncoksetmessages: + msg367744
2020-04-30 07:27:56hroncoksetnosy: + hroncok
messages: + msg367743
2020-04-30 03:14:18carljmsetmessages: + msg367730
2020-04-30 01:39:50gvanrossumsetmessages: + msg367726
2020-04-29 23:51:42Peter Ludemannsetmessages: + msg367716
2020-04-27 16:41:12Peter Ludemannsetnosy: + Peter Ludemann
2020-04-25 10:06:02corona10setpull_requests: - pull_request19032
2020-04-25 08:11:24corona10setnosy: + corona10
pull_requests: + pull_request19032
2020-04-24 22:55:35gregory.p.smithsetmessages: + msg367235
2020-04-24 21:15:50carljmsetmessages: + msg367230
2020-04-24 18:21:21gregory.p.smithsetmessages: + msg367209
2020-04-24 18:19:54gregory.p.smithsetmessages: + msg367208
2020-04-22 21:15:02carljmsetmessages: + msg367051
2020-04-22 20:50:05carljmsetpull_requests: + pull_request18987
2020-04-22 18:10:08eric.snowsetnosy: + eric.snow
2020-04-22 17:57:20BTaskayasetnosy: + BTaskaya
2020-04-22 17:31:39carljmsetnosy: + carljm
messages: + msg367031
2020-04-22 14:26:04gvanrossumsetnosy: + gvanrossum
messages: + msg367005
2020-04-22 05:01:15gregory.p.smithsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request18969
2020-04-22 04:41:28gregory.p.smithsetcomponents: + 2to3 (2.x to 3.x conversion tool)
stage: needs patch
2020-04-22 04:40:54gregory.p.smithcreate