classification
Title: Deprecate and remove distutils
Type: Stage:
Components: Distutils Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, christian.heimes, doko, dstufft, eric.araujo, hugovk, jaraco, lukasz.langa, ncoghlan, ned.deily, p-ganssle, pablogsal, paul.moore, steve.dower, yan12125
Priority: normal Keywords:

Created on 2020-07-12 08:30 by jaraco, last changed 2020-07-29 06:14 by hugovk.

Messages (16)
msg373548 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-07-12 08:30
Setuptools has adopted distutils as outlined in [pypa/packaging-problems#127](https://github.com/pypa/packaging-problems/issues/127). Although there are some straggling issues, the current release of Setuptools fully obviates distutils if a certain environment variable is set. Soon, that behavior will be default.

Additionally, the distutils codebase remains maintained at [pypa/distutils](https://github.com/pypa/distutils) in a form suitable for releasing as a third-party package, should the need arise (i.e. pip install distutils).

The plan now is to freeze, deprecate, and in Python N + 0.1, remove distutils.

Already, Setuptools is identifying emergent bugs and other defects in distutils and providing fixes for them (issue41207, [pypa/setuptools#2212](https://github.com/pypa/setuptools/issues/2212)). Keeping these changes in sync across three repos and different supported versions is tedious, so I'd like to move forward with the deprecation process as soon as possible.
msg373549 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-07-12 08:33
Łukasz, would it be possible to add the deprecation warning and documented deprecation to Python 3.9?
msg373586 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-07-13 07:28
So what is the plan to continue to support building cpython itself which depends on Distutils? Currently the build bootstraps itself without the aid of an existing Python interpreter instance. There would also be major impacts across the whole cpython development process. For example, there are many open Distutils issues in the bugs.python.org bug tracker. We need a plan on how those are to be handled (and that should take into account the expected transition from b.p.o to GitHub issues).  People will continue to submit issues agains Distutils there so triage team members and core developers need to know how to handle such issues.  What if an issue applies also or only to a previous release branch (i.e. where Distutils is still in the repo)?  What about Distutils documentation in the Python docset?  THose are just some off the top of my head.

I don't think any of these issues are necessarily blockers but they need to be planned for and reviewed.  I think a PEP is definitely in order for a change of this magnitude.
msg373612 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2020-07-13 16:43
It's too late to add a new deprecation in the Python 3.9 cycle. Next week is the *last* beta release. Most beta testing already took place.
msg373613 - (view) Author: Paul Ganssle (p-ganssle) * (Python committer) Date: 2020-07-13 16:50
> So what is the plan to continue to support building cpython itself which depends on Distutils? Currently the build bootstraps itself without the aid of an existing Python interpreter instance. There would also be major impacts across the whole cpython development process.

My understanding was that the plan was to move the standard library distutils into a private module somewhere in the standard library and presumably to slim it down to only the bare minimum required for what is necessary to build Python itself. We're really only concerned with the use of distutils to build packages.

> For example, there are many open Distutils issues in the bugs.python.org bug tracker. We need a plan on how those are to be handled (and that should take into account the expected transition from b.p.o to GitHub issues).  People will continue to submit issues agains Distutils there so triage team members and core developers need to know how to handle such issues. What if an issue applies also or only to a previous release branch (i.e. where Distutils is still in the repo)?

As far as I can tell we've already been telling people that issues in distutils should be fixed in setuptools instead for a few years. I don't think anything needs to be done about the currently open distutils tickets before we *deprecate* distutils, though during the deprecation period we'll probably want to decide whether we want to migrate them, do a mass closure or just leave them to be ad hoc closed as people stumble upon them later. Mass closure may be complicated because tickets affecting CPython itself will still need to be addressed.

> What about Distutils documentation in the Python docset?  THose are just some off the top of my head.

The distutils documentation is already basically just a warning page that says "stop using distutils": https://docs.python.org/3/library/distutils.html#module-distutils

Before these reference materials are removed from the docs we'll need to make sure that all the stuff that's still supported is documented on the setuptools side.

> I don't think any of these issues are necessarily blockers but they need to be planned for and reviewed.  I think a PEP is definitely in order for a change of this magnitude.

A PEP may be a good idea, but I do think the change doesn't have a particularly large magnitude. Anyone using setuptools or pip has already been getting setuptools' monkey-patched version of distutils for ages now, and soon they will be getting setuptools' vendored version. The documentation already indicates that distutils is at least soft-deprecated in favor of setuptools and we've already been directing issues and PRs to setuptools instead of distutils. This last piece is really formalizing something we've been incrementally working towards for a long time now. Doesn't mean we shouldn't do it carefully and with a lot of notice, but it's also not a sudden and massive shift.
msg373614 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-07-13 16:52
Deprecating in 3.10 is fine - everyone who needs to know about it releases whenever they like anyway, so we just need to make _some_ announcement.

I'd propose either moving it to Tools/distutils, or renaming it to _distutils. The point is that we're saying it's only fit for use for the core build now, and nobody else should ever import it (or complain about it ;) ).
msg373615 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2020-07-13 16:54
Maybe it would make sense to remove distutils from the name completely, _buildutils or something. Dunno, seems like it might be reasonable just to further separate it from the concept of "distutils" the public library.
msg373622 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2020-07-14 00:15
FYI PEP 387 (which I expect will be accepted once I catch up from vacation) specified deprecations are to be public for two releases before removal or approval from the SC for a shorter cycle.

So if distutils is deprecated in 3.10 then it can be removed in 3.12 or you can ask the SC for an exemption to do it in 3.11.
msg373629 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2020-07-14 08:48
> It's too late to add a new deprecation in the Python 3.9 cycle

Please can we add a note in 3.9, that it will be deprecated in 3.10?
msg373630 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2020-07-14 08:59
Renaming distutils to _buildutils only delays the problem to remove it.  But yes, it explicitly makes it explicit that code needs to be changed.

I would like to see that neither distutils or _buildutils is installed by default, and only is available internally for building the extensions of CPython.

The "old" build system to build builtins instead of extensions is still functional, so it should be ok to build the extensions also with the old build system.

That would require moving all the config stuff in setup.py to autoconf tests, which is perfectly doable.  The MacOS and Windows builds would need some attention too, but afaicr when asking Ned Deily and Steve Dower at the language summits, they didn't have a concern about this approach.
msg373631 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-07-14 09:15
+1

I would like to propose three changes:

1) rename distutils, either _distutils or _buildutils sounds good to me
2) make distutils a build-only dependency and no longer install it with make install and other install targets
3) start to build extensions from Makefile or Modules/Setup

For (3) we have to move some checks into autoconf and maybe extend Modules/Setup to support conditional compilation.
msg373633 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2020-07-14 09:39
> A PEP may be a good idea, but I do think the change doesn't have a
> particularly large magnitude. Anyone using setuptools or pip has
> already been getting setuptools' monkey-patched version of distutils
> for ages now, and soon they will be getting setuptools' vendored
> version. The documentation already indicates that distutils is at
> least soft-deprecated in favor of setuptools and we've already been
> directing issues and PRs to setuptools instead of distutils.

I don't think it's a good idea to replace bad habits from distutils with bad habits in setuptools._distutils.  And this is exactly what you get with pointing directly to setuptools.

While splitting out distutils to a separate package in a Linux distro, I found some creative usages at runtime of a package (see my lightning talk at the language summit 2018, and [1]).  From my point of view, CPython should provide documentation how to forward-port these issues without using setuptools._distutils.

Currently setuptools only has one component (pkg_resources, [2]) which is used at runtime.  I dislike it if more than that is used at runtime of a package.

[1] https://mail.python.org/archives/list/distutils-sig@python.org/thread/74WZ7D3ARF7B3N6MAV2JBV3DW6TRHFIV/
[2] https://github.com/pypa/setuptools/issues/863
msg373649 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-07-14 17:08
The Windows build doesn't depend on distutils at all. We've had dedicated build scripts for each module since before I started contributing.
msg373651 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-07-14 17:28
The Windows build system didn't use setu.py even before I upgrade the VS build system to VS 2010.
msg373652 - (view) Author: Paul Ganssle (p-ganssle) * (Python committer) Date: 2020-07-14 18:03
> I don't think it's a good idea to replace bad habits from distutils with bad habits in setuptools._distutils.  And this is exactly what you get with pointing directly to setuptools.

These are two different questions. We're not asking people to migrate to `setuptools._distutils` (a private module which may not continue to exist in that location), `setuptools` is *adopting* `distutils`, such that `distutils` is a project provided by `pip install distutils` (mind you, this is happening independent of what the standard library does — the only question is whether `import distutils` continues to work if you don't have `setuptools` installed).

> While splitting out distutils to a separate package in a Linux distro, I found some creative usages at runtime of a package (see my lightning talk at the language summit 2018, and [1]).  From my point of view, CPython should provide documentation how to forward-port these issues without using setuptools._distutils.

At this point, the extent of CPython's documentation on this should probably be, "We are removing `distutils` and moving it into the `setuptools` namespace. In future versions, you will need to install `setuptools` to import the `distutils` package." `setuptools` should almost certainly deprecate `distutils` and probably remove large swathes of it in the process, but that's probably on a case-by-case basis, and it's a separate issue from what needs to happen in CPython.

> Currently setuptools only has one component (pkg_resources, [2]) which is used at runtime.  I dislike it if more than that is used at runtime of a package.

I don't think anyone is planning to recommend the use of *any* `setuptools`-provided packages at runtime, including `pkg_resources`. This move is actually a good one from that point of view, because it will require that projects using `distutils` declare a *runtime* dependency on `setuptools`, which will, hopefully, raise some eyebrows. Better than the current situation, where these dependencies are totally undeclared (though probably worse than if `setuptools`, `pkg_resources` and `distutils` were all separate PyPI packages).
msg373653 - (view) Author: Paul Ganssle (p-ganssle) * (Python committer) Date: 2020-07-14 18:54
Oops, just realized my previous post said `pip install distutils`. I meant to say that `pip install setuptools` will provide the `distutils` module (right now you do `import setuptools; import distutils` and you get the setuptools-provided version; we're working on a version where `import distutils` comes from `setuptools` regardless of the import order).
History
Date User Action Args
2020-07-29 06:14:09hugovksetnosy: + hugovk
2020-07-18 05:23:52terry.reedysetversions: - Python 3.9
2020-07-14 18:54:46p-gansslesetmessages: + msg373653
2020-07-14 18:03:33p-gansslesetmessages: + msg373652
2020-07-14 17:28:49christian.heimessetmessages: + msg373651
2020-07-14 17:08:30steve.dowersetmessages: + msg373649
2020-07-14 10:57:31yan12125setnosy: + yan12125
2020-07-14 09:39:02dokosetmessages: + msg373633
2020-07-14 09:15:58christian.heimessetnosy: + christian.heimes
messages: + msg373631
2020-07-14 08:59:47dokosetmessages: + msg373630
2020-07-14 08:48:03dokosetnosy: + doko
messages: + msg373629
2020-07-14 00:15:28brett.cannonsetmessages: + msg373622
2020-07-13 17:43:52xtreaksetnosy: + p-ganssle
2020-07-13 16:54:44dstufftsetmessages: + msg373615
2020-07-13 16:52:31steve.dowersetnosy: - p-ganssle
messages: + msg373614
2020-07-13 16:50:52p-gansslesetnosy: + p-ganssle
messages: + msg373613
2020-07-13 16:43:11lukasz.langasetmessages: + msg373612
2020-07-13 07:29:37ned.deilysetnosy: + brett.cannon
2020-07-13 07:28:23ned.deilysetnosy: + ned.deily, pablogsal
messages: + msg373586
2020-07-12 08:50:51jaracosetnosy: + steve.dower
2020-07-12 08:34:02jaracosetnosy: + paul.moore, ncoghlan
2020-07-12 08:33:17jaracosetnosy: + lukasz.langa
messages: + msg373549
2020-07-12 08:30:25jaracocreate