Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compileall improvements #82293

Closed
encukou opened this issue Sep 11, 2019 · 11 comments
Closed

Compileall improvements #82293

encukou opened this issue Sep 11, 2019 · 11 comments
Labels
topic-installation

Comments

@encukou
Copy link
Member

encukou commented Sep 11, 2019

BPO 38112
Nosy @gpshead, @vstinner, @encukou, @frenzymadness, @pablogsal
PRs
  • bpo-38112: Compileall improvements #16012
  • bpo-38112: compileall: Skip long path path on Windows if the path can't be created #16414
  • bpo-38112: compileall tests: Use shorter name for long_path test #16419
  • bpo-38112: Document that compileall.compile_[dir,file] also accept multiple opt levels #20174
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2019-10-15.13:40:03.440>
created_at = <Date 2019-09-11.14:01:19.510>
labels = ['expert-installation']
title = 'Compileall improvements'
updated_at = <Date 2020-05-18.13:23:44.140>
user = 'https://github.com/encukou'

    bugs.python.org fields:

    activity = <Date 2020-05-18.13:23:44.140>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-10-15.13:40:03.440>
closer = 'vstinner'
components = ['Installation']
creation = <Date 2019-09-11.14:01:19.510>
creator = 'petr.viktorin'
dependencies = []
files = []
hgrepos = []
issue_num = 38112
keywords = ['patch']
message_count = 11.0
messages = ['351887', '353266', '353271', '353272', '353275', '353286', '353319', '354610', '354728', '362783', '369215']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'vstinner', 'petr.viktorin', 'frenzy', 'pablogsal']
pr_nums = ['16012', '16414', '16419', '20174']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue38112'
versions = []
    @encukou
    Copy link
    Member Author

    encukou commented Sep 11, 2019

    Hello,
    In Fedora, we've long used a Bash script to compile modules to bytecode, as the compileall module was historically unsuitable.
    Recently, Lumír re-wrote our enhancements to compileall, and published the result as compileall2 on PyPI. We'd like to get the improvements back into the standard library so everyone can benefit.
    The improvements are:

    • Raise the default recursion limit
    • Control of the path baked into the *.pyc file (-s to strip a prefix; -p to prepend a new one) to support different build/install locations.
    • Compiling for multiple optimization levels at once (by specifying -o multiple times)

    A PR should follow shortly.

    @encukou
    Copy link
    Member Author

    encukou commented Sep 26, 2019

    New changeset 8e7bb99 by Petr Viktorin (Lumír 'Frenzy' Balhar) in branch 'master':
    bpo-38112: Compileall improvements (GH-16012)
    8e7bb99

    @vstinner
    Copy link
    Member

    test_compileall failed on AMD64 Windows8.1 Non-Debug 3.x:
    https://buildbot.python.org/all/#/builders/12/builds/3257

    Example:

    ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\test\test_compileall.py", line 48, in setUp
    os.makedirs(self.long_path)
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  [Previous line repeated 18 more times]
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 221, in makedirs
    mkdir(name, mode)
FileNotFoundError: [WinError 206] The filename or extension is too long: 'D:\\temp\\tmpcgsk1z3q\\long\\1\\2\\3\\4\\5\\6\\7\\8\\9\\10\\11\\12\\13\\14\\15\\16\\17\\18\\19\\20\\21\\22\\23\\24\\25\\26\\27\\28\\29\\30\\31\\32\\33\\34\\35\\36\\37\\38\\39\\40\\41\\42\\43\\44\\45\\46\\47\\48\\49\\50\\51\\52\\53\\54\\55\\56\\57\\58\\59\\60\\61\\62\\63\\64\\65\\66\\67\\68\\69\\70\\71\\72\\73\\74\\75\\76\\77\\78'

    Failures:

    ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_files (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_path (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_magic_number (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_mtime (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_optimize (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_strip_only (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_compile_files (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_compile_path (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_magic_number (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_mtime (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_optimize (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_strip_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    FAIL: test_no_args_respects_force_flag (test.test_compileall.CommandLineTestsWithSourceEpoch)
    Re-running failed tests in verbose mode
    Re-running test_tarfile in verbose mode
    Re-running test_compileall in verbose mode
    ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_files (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_path (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_magic_number (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_mtime (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_optimize (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_strip_only (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
    ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_compile_files (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_compile_path (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_magic_number (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_mtime (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_optimize (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_strip_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
    ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)

    pythoninfo says:

    sys.windowsversion: sys.getwindowsversion(major=6, minor=3, build=9600, platform=2, service_pack='')
    windows.RtlAreLongPathsEnabled: <function not available>

    @vstinner
    Copy link
    Member

    Similar errors on AMD64 Windows10 3.x:
    https://buildbot.python.org/all/#/builders/3/builds/3567

    pythoninfo:

    sys.windowsversion: sys.getwindowsversion(major=10, minor=0, build=17134, platform=2, service_pack='')
    windows.RtlAreLongPathsEnabled: False

    --

    And AMD64 Windows7 SP1 3.x:
    https://buildbot.python.org/all/#/builders/40/builds/3180

    pythoninfo:

    sys.windowsversion: sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1')
    windows.RtlAreLongPathsEnabled: <function not available>

    @encukou
    Copy link
    Member Author

    encukou commented Sep 26, 2019

    Thanks!
    The buildbot complained in the pull request a while ago; I commented there that I'm looking into it.

    I'd appreciate a review for the fix; I don't have a good way to test it other than offer it to the buildbots.

    @encukou
    Copy link
    Member Author

    encukou commented Sep 26, 2019

    New changeset 4267c98 by Petr Viktorin in branch 'master':
    bpo-38112: compileall: Skip long path path on Windows if the path can't be created (GH-16414)
    4267c98

    @vstinner
    Copy link
    Member

    FYI test_compileall.test_compile_dir_maxlevels() also fails on AMD64 FreeBSD CURRENT Shared 3.x:
    https://buildbot.python.org/all/#builders/168/builds/1529

    @pablogsal
    Copy link
    Member

    The AMD64 Windows7 SP1 3.x buildbot has been failing for a while since commit 4267c98, please take a look at https://bugs.python.org/issue38470.

    @vstinner
    Copy link
    Member

    Should be fixed by:

    New changeset eb1dda2 by Petr Viktorin (Victor Stinner) in branch 'master':
    bpo-38470: Fix test_compileall.test_compile_dir_maxlevels() (GH-16789)
    eb1dda2

    @gpshead
    Copy link
    Member

    gpshead commented Feb 27, 2020

    https://bugs.python.org/issue39769 may be of interest. compileall's ddir= does not behave properly in recursive subdirectories, it did the right thing in 2.7 which is similar to a combination of compile_dir(dir, prependdir=ddir, stripdir=dir) after this issue's feature.

    @vstinner
    Copy link
    Member

    New changeset adc72bb by Lumír 'Frenzy' Balhar in branch 'master':
    bpo-38112: Document that compileall.compile_[dir,file] also accept multiple opt levels (GH-20174)
    adc72bb

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    topic-installation
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    4 participants
    @gpshead @vstinner @encukou @pablogsal