classification
Title: add the '--enable-sourceless-distribution' option to configure
Type: enhancement Stage: patch review
Components: Build, Cross-Build Versions: Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Alex.Willmer, barry, doko, koobs, martin.panter, ned.deily, r.david.murray, thomas-petazzoni, twouters, vstinner, xdegaye
Priority: normal Keywords: patch

Created on 2016-04-26 12:43 by xdegaye, last changed 2016-07-28 16:05 by xdegaye.

Files
File name Uploaded Description Edit
compileall-flags.patch xdegaye, 2016-04-26 12:43 review
compileall-flags_2.patch xdegaye, 2016-05-03 09:03 without the '-E' option review
legacy-pyc-files.patch xdegaye, 2016-07-28 10:31 review
Messages (17)
msg264272 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-04-26 12:43
Add a COMPILEALL_FLAGS Makefile variable to allow setting this flag to have legacy locations for byte-code files and save space on mobile devices.

Patch attached.
The '-E' python command line option added to $(PYTHON_FOR_BUILD) in the patch is fixing a problem that should actually be entered in another issue.  When cross-compiling, the PYTHON_FOR_BUILD command sets PYTHONPATH to the location of the cross-compiled shared libraries of the extension modules.  So the compilation of modules that import extension modules fail without '-E'.
msg264699 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-03 09:03
The python '-E' option was needed on 3.4 when this patch was first implemented and is not needed anymore at the current tip (changeset 2ef61a4747eb, Apr 27 2016).

New patch.
msg270920 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-07-21 13:24
> The python '-E' option was needed on 3.4 when this patch was first implemented and is not needed anymore

See msg 269359 in issue 22724.
msg271515 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-07-28 10:31
Install byte-code files to their legacy locations and names to save space (default are the PEP 3147 locations and names) when configure is run with '--enable-legacy-pyc-files'.

The patch does not prevent ensurepip to use PEP 3147 locations and names when it is run at the end of the installation.

As many tests use the linecache module, it makes sense to skip the installation of the test suite aas well since the patch removes all *.py files (except those installed by ensurepip in site-packages). This can be done when issue 27640 is resolved by using '--disable-test-suite'.

The size of the standard library [1] [2]:
    plain install: 111M
    --disable-test-suite: 53M
    --enable-legacy-pyc-files --disable-test-suite: 23M
    --enable-legacy-pyc-files --disable-test-suite --with-ensurepip=no: 14M

[1] without the extension modules
[2] excluding the LIBPL directory that is installed at --prefix instead of --exec-prefix for some reason and that contains miscellaneous stuff needed for extending/embedding. This is not needed on a mobile device that does not have any build tool.
msg271517 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2016-07-28 10:48
hmm, I really don't buy the space-saving argument.  you are saving some space with shorter path names, nothing more. so why do you introduce this option?
msg271520 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-07-28 11:12
> hmm, I really don't buy the space-saving argument.  you are saving some space with shorter path names, nothing more. so why do you introduce this option?

No, compileall is run with '-b', so there are no  PEP 3147 __pycache__ directories, and with an installation of 53M where the test suite is excluded, that saves 30M (see my previous msg).

IMHO this option is very useful on mobile devices and embedded systems where space is sparse.
msg271524 - (view) Author: Thomas Petazzoni (thomas-petazzoni) Date: 2016-07-28 11:18
I can also say that in the Buildroot project, we have patches to get rid of the PEP3147 stuff. Indeed, the PEP3147 forces one to have both the .py *and* the .pyc file for a given module. If you have only the .pyc file, Python does not recognize it and it cannot be imported.

By removing the PEP3147 functionality, we are able to keep only the .pyc files on the target, therefore dividing roughly by two the size of the Python installation.
msg271525 - (view) Author: Thomas Petazzoni (thomas-petazzoni) Date: 2016-07-28 11:18
See https://git.buildroot.org/buildroot/tree/package/python3/0016-Add-importlib-fix-for-PEP-3147-issue.patch
msg271527 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-07-28 11:23
I think a proposal to add an option like this requires more discussion, probably a PEP.
msg271528 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2016-07-28 11:24
but these are rebuilt when you start the interpreter, aren't they?
msg271529 - (view) Author: Thomas Petazzoni (thomas-petazzoni) Date: 2016-07-28 11:28
No, if you remove PEP3147 (like the Buildroot patch does), and install only the .pyc files, you have a smaller Python installation, and nothing gets "re-generated", since what you already have are the .pyc files. The .py files are not even installed on the target system.
msg271531 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-07-28 11:52
> but these are rebuilt when you start the interpreter, aren't they?

No.

Quoting PEP 3147:
    Case 4: legacy pyc files and source-less imports

    Python will ignore all legacy pyc files when a source file exists next to it. In other words, if a foo.pyc file exists next to the foo.py file, the pyc file will be ignored in all cases

    In order to continue to support source-less distributions though, if the source file is missing, Python will import a lone pyc file if it lives where the source file would have been.
msg271535 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-07-28 12:13
Our official position (so far, mobile is a new use case since the last time it was discussed) is that we don't go out of our way to support sourceless distribution, and in general we discourage it.  That is, it is left to the packager of a sourceless program to deal with removing the source and putting the .pyc files in the right place for python to find them.  So, yes, this requires at *least* a discussion on python-ideas, and possibly a PEP.

If this is adopted '--enable-legacy-pyc' would be the wrong name for the option, since it is in fact forcing legacy pyc...and if the only reason is to reduce space by omitting the source, then it should also not install the source so the name is wrong :)

But, this conversation should continue on python-ideas.  I'm -1 myself, for the record.
msg271538 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-07-28 12:20
About doko's note and your response: unless this patch does not install the source (I haven't looked), the __pycache__ files *will* be rebuilt at interpreter start, according to the text you quoted.
msg271544 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-07-28 13:15
Also, has anyone tried bundling the std lib into a zlib?  I know that option has been around for a long time but I don't know if it still works or is even being used.  Presumably, that would be another way to save space and file system entries.
msg271562 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-07-28 16:01
> and if the only reason is to reduce space by omitting the source, then it should also not install the source so the name is wrong :)

I miss your point, msg271515  said earlier "the patch removes all *.py files".
The patch does a sourceless distribution: no source and no __pycache__ files (with the exception mentionned in msg271515).
Changing the issue title with '--enable-sourceless-distribution' which is a better name and would have maybe avoided this confusion :)
msg271563 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-07-28 16:05
> About doko's note and your response: unless this patch does not install the source (I haven't looked), the __pycache__ files *will* be rebuilt at interpreter start, according to the text you quoted.

Right, and since the source files are removed, the __pycache__ files are *not* built. Hence my negative answer to Matthias.
History
Date User Action Args
2017-12-10 09:30:59xdegayeunlinkissue26865 dependencies
2016-07-28 16:05:00xdegayesetmessages: + msg271563
2016-07-28 16:01:24xdegayesetmessages: + msg271562
title: add the '--enable-legacy-pyc-files' option to configure -> add the '--enable-sourceless-distribution' option to configure
2016-07-28 13:15:56ned.deilysetmessages: + msg271544
2016-07-28 12:20:20r.david.murraysetmessages: + msg271538
2016-07-28 12:13:23r.david.murraysetnosy: + r.david.murray
messages: + msg271535
2016-07-28 11:52:18xdegayesetmessages: + msg271531
2016-07-28 11:28:39thomas-petazzonisetmessages: + msg271529
2016-07-28 11:24:19dokosetmessages: + msg271528
2016-07-28 11:23:21ned.deilysetnosy: + ned.deily
messages: + msg271527
2016-07-28 11:20:24koobssetnosy: + koobs
2016-07-28 11:18:44thomas-petazzonisetmessages: + msg271525
2016-07-28 11:18:08thomas-petazzonisetmessages: + msg271524
2016-07-28 11:12:53xdegayesetmessages: + msg271520
2016-07-28 10:48:26dokosetnosy: + barry
2016-07-28 10:48:14dokosetmessages: + msg271517
2016-07-28 10:47:45xdegayesetnosy: + vstinner, thomas-petazzoni
2016-07-28 10:31:17xdegayesetfiles: + legacy-pyc-files.patch

components: + Build
title: android: add a COMPILEALL_FLAGS Makefile variable -> add the '--enable-legacy-pyc-files' option to configure
nosy: + doko, martin.panter

messages: + msg271515
stage: patch review
2016-07-21 13:24:17xdegayesetmessages: + msg270920
2016-05-03 09:03:41xdegayesetfiles: + compileall-flags_2.patch

messages: + msg264699
2016-05-03 07:18:37xdegayesettitle: add a COMPILEALL_FLAGS Makefile variable -> android: add a COMPILEALL_FLAGS Makefile variable
2016-05-01 07:12:24xdegayesetnosy: + twouters
2016-04-26 16:04:41zach.warelinkissue26865 dependencies
2016-04-26 12:43:31xdegayecreate