classification
Title: Provide configure options to enable/disable Python modules and extensions
Type: enhancement Stage:
Components: Build Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, BreamoreBoy, doko, eric.araujo, haypo, koobs, loewis, pitrou, thomas-petazzoni
Priority: normal Keywords: patch

Created on 2014-01-09 22:52 by thomas-petazzoni, last changed 2014-03-26 11:31 by loewis.

Files
File name Uploaded Description Edit
0001-Add-infrastructure-to-disable-the-build-of-certain-e.patch thomas-petazzoni, 2014-01-09 22:53 review
0002-Add-an-option-to-disable-installation-of-test-module.patch thomas-petazzoni, 2014-01-09 22:53
0003-Add-an-option-to-disable-pydoc.patch thomas-petazzoni, 2014-01-09 22:53
0004-Add-an-option-to-disable-lib2to3.patch thomas-petazzoni, 2014-01-09 22:53
0005-Add-option-to-disable-the-sqlite3-module.patch thomas-petazzoni, 2014-01-09 22:53
0006-Add-an-option-to-disable-the-tk-module.patch thomas-petazzoni, 2014-01-09 22:53
0007-Add-an-option-to-disable-the-curses-module.patch thomas-petazzoni, 2014-01-09 22:54
0008-Add-an-option-to-disable-expat.patch thomas-petazzoni, 2014-01-09 22:54
0009-Add-an-option-to-disable-CJK-codecs.patch thomas-petazzoni, 2014-01-09 22:54
0010-Add-an-option-to-disable-NIS.patch thomas-petazzoni, 2014-01-09 22:54
0011-Add-an-option-to-disable-unicodedata.patch thomas-petazzoni, 2014-01-09 22:54
0012-Add-an-option-to-disable-IDLE.patch thomas-petazzoni, 2014-01-09 22:54
Messages (16)
msg207802 - (view) Author: Thomas Petazzoni (thomas-petazzoni) Date: 2014-01-09 22:52
In the context of space-constrained embedded Linux systems, installing the entire set of Python modules and extensions is not necessarily desirable. For example, all the test modules, as well as certain extensions requiring third-parties libraries are often unnecessary, and uselessly consume precious storage on the embedded Linux system.

While we could certainly remove these undesired modules and extensions manually, it is much more convenient to have configuration options to selectively enable and disable them. Another very strong benefit of having configuration options is that we can actually *disable* the build of these unneeded modules and extensions, therefore saving a lot of build time, which is very nice when you're repeatedly cross-compiling an entire embedded Linux system.

The proposed set of patches add several --enable-<foo>/--disable-<foo> options to enable/disable certain Python modules and extensions. These patches have been part of Buildroot, an embedded Linux build system (used for example by Google, and many embedded processor vendors, as well as a huge number of embedded system makers) for a while, and are useful to all our users using Python on their embedded Linux systems. Instead of carrying them around, we would like to have them merged in upstream Python.

Of course, we are definitely open to discussion on the approach to take to implement this configurability, and I'm ready to rework the patches according to the comments received here.

Thanks!
msg214742 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-03-24 21:50
I don't really like the idea of complicating our build tools even more. Can't you simply prune the install tree yourself?
msg214746 - (view) Author: Thomas Petazzoni (thomas-petazzoni) Date: 2014-03-24 22:09
No, because it's not simply about the size of the installed Python standard library: it's also about the number of dependencies to build before being able to build Python. For example, a normal Python installation requires OpenSSL, libncurses, and lots of other things. On many embedded systems, those are not needed.
msg214747 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-03-24 22:11
> For example, a normal Python installation requires OpenSSL,
> libncurses, and lots of other things.

Not really. If some development libraries are not available, Python should still install fine without the corresponding modules.
msg214749 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2014-03-24 22:23
The main issue with the proposed changes is that it redefines what “the Python standard library” is.  Right now, users can mostly expect modules listed in the official Python docs to be available in their installation, regardless of how they got their Python.

I say “mostly” because a distributor may exclude tests from a binay package, split some extensions modules like _tkinter and _lzma in other packages, etc.  A big source of pain in the past was distributors splitting distutils, but I think they all stopped.

(A related issue is that some distributors backport many bug fixes and sometimes features, which is a pain when you think you’ve run your tests with “Python X.Y.Z” and it was actually “X.Y.Z+some-patches”.)
msg214751 - (view) Author: Thomas Petazzoni (thomas-petazzoni) Date: 2014-03-24 22:32
But this expectation is not true: if dependencies are not available, Python silently disables the build of certain modules. So this story of making the standard library always has all the modules does not really stand.
msg214782 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2014-03-25 08:14
> But this expectation is not true: if dependencies are not available,
 > Python silently disables the build of certain modules. So this story
> of making the standard library always has all the modules does not really stand.

I think this one is a valid concern. Did run into it myself, because a final 
package was built on another kernel and disabled some semaphore stuff by accident.

None of the extensions above are built as builtins by default, so you can always 
prune these after the build.  What I think is needed is a mode which breaks the 
build when some extensions are not built. Whether this should be the default or 
not, ... but it would give you a chance for a deterministic build.  And the 
build system already tells you at the end of the extension builds that some 
extensions didn't build which were expected to build.
msg214792 - (view) Author: koobs (koobs) Date: 2014-03-25 08:42
These are a good step toward bringing the otherwise neglected Python build system back to the real world in terms of standard functionality, and will among other things, make life an absolute pleasure for downstreams and users alike.

"User-Serviceable" options are expected (in particular in autotools-based build systems), *not* a luxury, and have been missing from the start.

A complicated build system is not a function its feature-set or flexibility, but of the quality of its evolution.

There is also a distinction between the ability to customise the options of a build, and the defaults of those options. "Will no longer be a standard library" is a straw man.

These patches present only as a user-configurable extension to otherwise statically defined configurations that must be patched manually to modify. This is painful.

With my downstream (FreeBSD) porter & consumer-and-hacker-of-Python-build-mechanics hat on, I'd like to see these and more 'options' out-of-the-box.
msg214809 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-03-25 10:52
> A complicated build system is not a function its feature-set or
> flexibility, but of the quality of its evolution.

Certainly, but that doesn't change the concrete issue: we have a complicated build system that these patches will make more complicated.
msg214828 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-03-25 14:48
I certainly like the principle.  Does this need a wider audience, python-dev maybe?
msg214883 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2014-03-26 08:55
If it is planned to support BSD make, then partial rewrite of patches will be needed.

Example of syntax of GNU make:
ifeq (something,something)
…
endif

Example of syntax of FreeBSD make:
.if ${variable}==something
…
.endif
msg214884 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2014-03-26 09:16
According to koobs, building of CPython with FreeBSD make works at least with -j1 (and sporadically fails with higher value).
msg214885 - (view) Author: koobs (koobs) Date: 2014-03-26 09:18
More precisely:

Python 3.3 fails at anything > -j1 (switching to gmake makes this go away)
Python 3.4 has not failed up to -j8 (with bsd make)
msg214888 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-03-26 10:33
Antoine Pitrou wrote:
"I don't really like the idea of complicating our build tools even more. Can't you simply prune the install tree yourself?"

In the embedded world, the (cross) compilation process is very complex and slow. Being able to disable features makes this task simpler.

Éric Araujo wrote: 
"The main issue with the proposed changes is that it redefines what “the Python standard library” is."

I disagree. It's a common practice that a vendor gives the user the choice to enable or disable some features. On Gentoo and FreeBSD, you can disable features like IPv6 or shared memory on some packages. I don't think that it's currently possible on Python, but I would not be surprised to be able to enable or disable some features.

Éric Araujo wrote: 
"Right now, users can mostly expect modules listed in the official Python docs to be available in their installation, regardless of how they got their Python."

How you get Python matters :-) Python documentations describes the vanilla flavor distributed at python.org. But the Python license allows to strip some features without changing the name of the Python.

--

0002-Add-an-option-to-disable-installation-of-test-module.patch is interested. I never understand why Python installs its test suite. Who use this test suite installed on the system? Maybe the packager of the module to test Python. Ok, but the test suite can then be removed.

I like the overall approach, by individual patches may be discussed. For example,  0008-Add-an-option-to-disable-expat.patch breaks Python XML modules. Are they still be installed? I mean the modules implemented in Python and relying on the expat Python module.

The changes should be be documented somewhere. In the Python documentation, or at least in the "devguide".
msg214892 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-03-26 10:39
> Antoine Pitrou wrote:
> "I don't really like the idea of complicating our build tools even
> more. Can't you simply prune the install tree yourself?"
> 
> In the embedded world, the (cross) compilation process is very complex
> and slow. Being able to disable features makes this task simpler.

That's not really the point. The question is why we should have to
maintain this ourselves. It is easy for interested people to maintain
their own forks, especially when *removing* stuff.

For the record, we don't have a single cross-compiling buildbot: it
isn't a supported setup.

> 0002-Add-an-option-to-disable-installation-of-test-module.patch is
> interested. I never understand why Python installs its test suite. Who
> use this test suite installed on the system? Maybe the packager of the
> module to test Python. Ok, but the test suite can then be removed.

How else do you want to test that your Python installation works, other
than running the test suite?
msg214896 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-03-26 11:31
I'm +1 on the general idea, but -1 on the implementation strategy used.

Instead of coming up with configure options for selected (apparently problematic) modules, I'd like to see a solution that covers *all* extension modules.

One approach could be to reserve the option prefix --enable-mod-XXX for this kind of configuration, allowing people to specify --disable-mod-zipimport (for example).

Another approach (closer to what we already have) would be to support a *disabled* marker in Modules/Setup (and Modules/Setup.local), so anybody wishing to disable modules could put

*disabled*
zipimport
_sre
unicodedata

into Modules/Setup.local (rather than specifying it on the configure command line).
History
Date User Action Args
2014-03-26 11:31:34loewissetmessages: + msg214896
2014-03-26 10:39:37pitrousetnosy: + loewis
2014-03-26 10:39:12pitrousetmessages: + msg214892
2014-03-26 10:33:31hayposetmessages: + msg214888
2014-03-26 09:18:19koobssetmessages: + msg214885
2014-03-26 09:16:42Arfreversetmessages: + msg214884
2014-03-26 08:55:24Arfreversetnosy: + Arfrever
messages: + msg214883
2014-03-25 14:48:24BreamoreBoysetnosy: + BreamoreBoy
messages: + msg214828
2014-03-25 10:52:21pitrousetmessages: + msg214809
2014-03-25 08:42:11koobssetmessages: + msg214792
2014-03-25 08:14:11dokosetmessages: + msg214782
2014-03-24 22:32:01thomas-petazzonisetmessages: + msg214751
2014-03-24 22:23:46eric.araujosetnosy: + eric.araujo
messages: + msg214749
2014-03-24 22:11:28pitrousetmessages: + msg214747
2014-03-24 22:09:03thomas-petazzonisetmessages: + msg214746
2014-03-24 21:50:31pitrousetnosy: + pitrou
messages: + msg214742
2014-03-24 11:41:37koobssetnosy: + koobs
2014-02-17 23:07:32hayposetnosy: + doko, haypo
2014-01-09 22:54:48thomas-petazzonisetfiles: + 0012-Add-an-option-to-disable-IDLE.patch
2014-01-09 22:54:41thomas-petazzonisetfiles: + 0011-Add-an-option-to-disable-unicodedata.patch
2014-01-09 22:54:35thomas-petazzonisetfiles: + 0010-Add-an-option-to-disable-NIS.patch
2014-01-09 22:54:28thomas-petazzonisetfiles: + 0009-Add-an-option-to-disable-CJK-codecs.patch
2014-01-09 22:54:17thomas-petazzonisetfiles: + 0008-Add-an-option-to-disable-expat.patch
2014-01-09 22:54:04thomas-petazzonisetfiles: + 0007-Add-an-option-to-disable-the-curses-module.patch
2014-01-09 22:53:55thomas-petazzonisetfiles: + 0006-Add-an-option-to-disable-the-tk-module.patch
2014-01-09 22:53:47thomas-petazzonisetfiles: + 0005-Add-option-to-disable-the-sqlite3-module.patch
2014-01-09 22:53:33thomas-petazzonisetfiles: + 0004-Add-an-option-to-disable-lib2to3.patch
2014-01-09 22:53:25thomas-petazzonisetfiles: + 0003-Add-an-option-to-disable-pydoc.patch
2014-01-09 22:53:16thomas-petazzonisetfiles: + 0002-Add-an-option-to-disable-installation-of-test-module.patch
2014-01-09 22:53:04thomas-petazzonisetfiles: + 0001-Add-infrastructure-to-disable-the-build-of-certain-e.patch
keywords: + patch
2014-01-09 22:52:52thomas-petazzonisettype: enhancement
2014-01-09 22:52:42thomas-petazzonicreate