Add a COMPILEALL_FLAGS Makefile variable to allow setting this flag to have legacy locations for byte-code files and save space on mobile devices.

Patch attached.
The '-E' python command line option added to $(PYTHON_FOR_BUILD) in the patch is fixing a problem that should actually be entered in another issue.  When cross-compiling, the PYTHON_FOR_BUILD command sets PYTHONPATH to the location of the cross-compiled shared libraries of the extension modules.  So the compilation of modules that import extension modules fail without '-E'.

The python '-E' option was needed on 3.4 when this patch was first implemented and is not needed anymore at the current tip (changeset 2ef61a4747eb, Apr 27 2016).

New patch.

> The python '-E' option was needed on 3.4 when this patch was first implemented and is not needed anymore

See msg 269359 in issue 22724.

Install byte-code files to their legacy locations and names to save space (default are the PEP 3147 locations and names) when configure is run with '--enable-legacy-pyc-files'.

The patch does not prevent ensurepip to use PEP 3147 locations and names when it is run at the end of the installation.

As many tests use the linecache module, it makes sense to skip the installation of the test suite aas well since the patch removes all *.py files (except those installed by ensurepip in site-packages). This can be done when issue 27640 is resolved by using '--disable-test-suite'.

The size of the standard library [1] [2]:
    plain install: 111M
    --disable-test-suite: 53M
    --enable-legacy-pyc-files --disable-test-suite: 23M
    --enable-legacy-pyc-files --disable-test-suite --with-ensurepip=no: 14M

[1] without the extension modules
[2] excluding the LIBPL directory that is installed at --prefix instead of --exec-prefix for some reason and that contains miscellaneous stuff needed for extending/embedding. This is not needed on a mobile device that does not have any build tool.

hmm, I really don't buy the space-saving argument.  you are saving some space with shorter path names, nothing more. so why do you introduce this option?

> hmm, I really don't buy the space-saving argument.  you are saving some space with shorter path names, nothing more. so why do you introduce this option?

No, compileall is run with '-b', so there are no  PEP 3147 __pycache__ directories, and with an installation of 53M where the test suite is excluded, that saves 30M (see my previous msg).

IMHO this option is very useful on mobile devices and embedded systems where space is sparse.

I can also say that in the Buildroot project, we have patches to get rid of the PEP3147 stuff. Indeed, the PEP3147 forces one to have both the .py *and* the .pyc file for a given module. If you have only the .pyc file, Python does not recognize it and it cannot be imported.

By removing the PEP3147 functionality, we are able to keep only the .pyc files on the target, therefore dividing roughly by two the size of the Python installation.

See https://git.buildroot.org/buildroot/tree/package/python3/0016-Add-importlib-fix-for-PEP-3147-issue.patch

I think a proposal to add an option like this requires more discussion, probably a PEP.

but these are rebuilt when you start the interpreter, aren't they?

No, if you remove PEP3147 (like the Buildroot patch does), and install only the .pyc files, you have a smaller Python installation, and nothing gets "re-generated", since what you already have are the .pyc files. The .py files are not even installed on the target system.

> but these are rebuilt when you start the interpreter, aren't they?

No.

Quoting PEP 3147:
    Case 4: legacy pyc files and source-less imports

    Python will ignore all legacy pyc files when a source file exists next to it. In other words, if a foo.pyc file exists next to the foo.py file, the pyc file will be ignored in all cases

    In order to continue to support source-less distributions though, if the source file is missing, Python will import a lone pyc file if it lives where the source file would have been.

Our official position (so far, mobile is a new use case since the last time it was discussed) is that we don't go out of our way to support sourceless distribution, and in general we discourage it.  That is, it is left to the packager of a sourceless program to deal with removing the source and putting the .pyc files in the right place for python to find them.  So, yes, this requires at *least* a discussion on python-ideas, and possibly a PEP.

If this is adopted '--enable-legacy-pyc' would be the wrong name for the option, since it is in fact forcing legacy pyc...and if the only reason is to reduce space by omitting the source, then it should also not install the source so the name is wrong :)

But, this conversation should continue on python-ideas.  I'm -1 myself, for the record.

About doko's note and your response: unless this patch does not install the source (I haven't looked), the __pycache__ files *will* be rebuilt at interpreter start, according to the text you quoted.

Also, has anyone tried bundling the std lib into a zlib?  I know that option has been around for a long time but I don't know if it still works or is even being used.  Presumably, that would be another way to save space and file system entries.

> and if the only reason is to reduce space by omitting the source, then it should also not install the source so the name is wrong :)

I miss your point, msg271515  said earlier "the patch removes all *.py files".
The patch does a sourceless distribution: no source and no __pycache__ files (with the exception mentionned in msg271515).
Changing the issue title with '--enable-sourceless-distribution' which is a better name and would have maybe avoided this confusion :)

> About doko's note and your response: unless this patch does not install the source (I haven't looked), the __pycache__ files *will* be rebuilt at interpreter start, according to the text you quoted.

Right, and since the source files are removed, the __pycache__ files are *not* built. Hence my negative answer to Matthias.