Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add the '--enable-sourceless-distribution' option to configure #71039

Open
xdegaye mannequin opened this issue Apr 26, 2016 · 17 comments
Open

add the '--enable-sourceless-distribution' option to configure #71039

xdegaye mannequin opened this issue Apr 26, 2016 · 17 comments
Labels
build The build process and cross-build type-feature A feature request or enhancement

Comments

@xdegaye
Copy link
Mannequin

xdegaye mannequin commented Apr 26, 2016

BPO 26852
Nosy @Yhg1s, @warsaw, @doko42, @vstinner, @ned-deily, @mcepl, @bitdancer, @vadmium, @koobs, @tpetazzoni, @moreati
Files
  • compileall-flags.patch
  • compileall-flags_2.patch: without the '-E' option
  • legacy-pyc-files.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2016-04-26.12:43:31.203>
labels = ['type-feature', 'build']
title = "add the '--enable-sourceless-distribution' option to configure"
updated_at = <Date 2019-12-10.08:08:26.308>
user = 'https://github.com/xdegaye'

    bugs.python.org fields:

    activity = <Date 2019-12-10.08:08:26.308>
actor = 'xdegaye'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build', 'Cross-Build']
creation = <Date 2016-04-26.12:43:31.203>
creator = 'xdegaye'
dependencies = []
files = ['42603', '42690', '43919']
hgrepos = []
issue_num = 26852
keywords = ['patch']
message_count = 17.0
messages = ['264272', '264699', '270920', '271515', '271517', '271520', '271524', '271525', '271527', '271528', '271529', '271531', '271535', '271538', '271544', '271562', '271563']
nosy_count = 11.0
nosy_names = ['twouters', 'barry', 'doko', 'vstinner', 'ned.deily', 'mcepl', 'r.david.murray', 'martin.panter', 'koobs', 'thomas-petazzoni', 'Alex.Willmer']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26852'
versions = ['Python 3.6']
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Apr 26, 2016

    Add a COMPILEALL_FLAGS Makefile variable to allow setting this flag to have legacy locations for byte-code files and save space on mobile devices.

    Patch attached.
    The '-E' python command line option added to $(PYTHON_FOR_BUILD) in the patch is fixing a problem that should actually be entered in another issue. When cross-compiling, the PYTHON_FOR_BUILD command sets PYTHONPATH to the location of the cross-compiled shared libraries of the extension modules. So the compilation of modules that import extension modules fail without '-E'.

    @xdegaye xdegaye mannequin added build The build process and cross-build type-feature A feature request or enhancement labels Apr 26, 2016
    @xdegaye xdegaye mannequin changed the title add a COMPILEALL_FLAGS Makefile variable android: add a COMPILEALL_FLAGS Makefile variable May 3, 2016
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 3, 2016

    The python '-E' option was needed on 3.4 when this patch was first implemented and is not needed anymore at the current tip (changeset 2ef61a4747eb, Apr 27 2016).

    New patch.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Jul 21, 2016

    The python '-E' option was needed on 3.4 when this patch was first implemented and is not needed anymore

    See msg 269359 in bpo-22724.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Jul 28, 2016

    Install byte-code files to their legacy locations and names to save space (default are the PEP-3147 locations and names) when configure is run with '--enable-legacy-pyc-files'.

    The patch does not prevent ensurepip to use PEP-3147 locations and names when it is run at the end of the installation.

    As many tests use the linecache module, it makes sense to skip the installation of the test suite aas well since the patch removes all *.py files (except those installed by ensurepip in site-packages). This can be done when bpo-27640 is resolved by using '--disable-test-suite'.

    The size of the standard library [1] [2]:
    plain install: 111M
    --disable-test-suite: 53M
    --enable-legacy-pyc-files --disable-test-suite: 23M
    --enable-legacy-pyc-files --disable-test-suite --with-ensurepip=no: 14M

    [1] without the extension modules
    [2] excluding the LIBPL directory that is installed at --prefix instead of --exec-prefix for some reason and that contains miscellaneous stuff needed for extending/embedding. This is not needed on a mobile device that does not have any build tool.

    @xdegaye xdegaye mannequin added the build The build process and cross-build label Jul 28, 2016
    @xdegaye xdegaye mannequin changed the title android: add a COMPILEALL_FLAGS Makefile variable add the '--enable-legacy-pyc-files' option to configure Jul 28, 2016
    @doko42
    Copy link
    Member

    doko42 commented Jul 28, 2016

    hmm, I really don't buy the space-saving argument. you are saving some space with shorter path names, nothing more. so why do you introduce this option?

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Jul 28, 2016

    hmm, I really don't buy the space-saving argument. you are saving some space with shorter path names, nothing more. so why do you introduce this option?

    No, compileall is run with '-b', so there are no PEP-3147 __pycache__ directories, and with an installation of 53M where the test suite is excluded, that saves 30M (see my previous msg).

    IMHO this option is very useful on mobile devices and embedded systems where space is sparse.

    @tpetazzoni
    Copy link
    Mannequin

    tpetazzoni mannequin commented Jul 28, 2016

    I can also say that in the Buildroot project, we have patches to get rid of the PEP-3147 stuff. Indeed, the PEP-3147 forces one to have both the .py *and* the .pyc file for a given module. If you have only the .pyc file, Python does not recognize it and it cannot be imported.

    By removing the PEP-3147 functionality, we are able to keep only the .pyc files on the target, therefore dividing roughly by two the size of the Python installation.

    @tpetazzoni
    Copy link
    Mannequin

    tpetazzoni mannequin commented Jul 28, 2016

    See https://git.buildroot.org/buildroot/tree/package/python3/0016-Add-importlib-fix-for-PEP-3147-issue.patch

    @ned-deily
    Copy link
    Member

    I think a proposal to add an option like this requires more discussion, probably a PEP.

    @doko42
    Copy link
    Member

    doko42 commented Jul 28, 2016

    but these are rebuilt when you start the interpreter, aren't they?

    @tpetazzoni
    Copy link
    Mannequin

    tpetazzoni mannequin commented Jul 28, 2016

    No, if you remove PEP-3147 (like the Buildroot patch does), and install only the .pyc files, you have a smaller Python installation, and nothing gets "re-generated", since what you already have are the .pyc files. The .py files are not even installed on the target system.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Jul 28, 2016

    but these are rebuilt when you start the interpreter, aren't they?

    No.

    Quoting PEP-3147:
    Case 4: legacy pyc files and source-less imports

    Python will ignore all legacy pyc files when a source file exists next to it. In other words, if a foo.pyc file exists next to the foo.py file, the pyc file will be ignored in all cases

In order to continue to support source-less distributions though, if the source file is missing, Python will import a lone pyc file if it lives where the source file would have been.

    @bitdancer
    Copy link
    Member

    Our official position (so far, mobile is a new use case since the last time it was discussed) is that we don't go out of our way to support sourceless distribution, and in general we discourage it. That is, it is left to the packager of a sourceless program to deal with removing the source and putting the .pyc files in the right place for python to find them. So, yes, this requires at *least* a discussion on python-ideas, and possibly a PEP.

    If this is adopted '--enable-legacy-pyc' would be the wrong name for the option, since it is in fact forcing legacy pyc...and if the only reason is to reduce space by omitting the source, then it should also not install the source so the name is wrong :)

    But, this conversation should continue on python-ideas. I'm -1 myself, for the record.

    @bitdancer
    Copy link
    Member

    About doko's note and your response: unless this patch does not install the source (I haven't looked), the __pycache__ files *will* be rebuilt at interpreter start, according to the text you quoted.

    @ned-deily
    Copy link
    Member

    Also, has anyone tried bundling the std lib into a zlib? I know that option has been around for a long time but I don't know if it still works or is even being used. Presumably, that would be another way to save space and file system entries.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Jul 28, 2016

    and if the only reason is to reduce space by omitting the source, then it should also not install the source so the name is wrong :)

    I miss your point, msg271515 said earlier "the patch removes all *.py files".
    The patch does a sourceless distribution: no source and no __pycache__ files (with the exception mentionned in msg271515).
    Changing the issue title with '--enable-sourceless-distribution' which is a better name and would have maybe avoided this confusion :)

    @xdegaye xdegaye mannequin changed the title add the '--enable-legacy-pyc-files' option to configure add the '--enable-sourceless-distribution' option to configure Jul 28, 2016
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Jul 28, 2016

    About doko's note and your response: unless this patch does not install the source (I haven't looked), the __pycache__ files *will* be rebuilt at interpreter start, according to the text you quoted.

    Right, and since the source files are removed, the __pycache__ files are *not* built. Hence my negative answer to Matthias.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    build The build process and cross-build type-feature A feature request or enhancement
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @bitdancer @ned-deily @doko42