➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Compileall improvements
Type: Stage: resolved
Components: Installation Versions:
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: frenzy, gregory.p.smith, pablogsal, petr.viktorin, vstinner
Priority: normal Keywords: patch

Created on 2019-09-11 14:01 by petr.viktorin, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 16012 merged frenzy, 2019-09-12 04:54
PR 16414 merged petr.viktorin, 2019-09-26 07:19
PR 16419 merged petr.viktorin, 2019-09-26 11:56
PR 20174 merged frenzy, 2020-05-18 13:23
Messages (11)
msg351887 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-09-11 14:01 
Hello,
In Fedora, we've long used a Bash script to compile modules to bytecode, as the compileall module was historically unsuitable.
Recently, Lumír re-wrote our enhancements to compileall, and published the result as compileall2 on PyPI. We'd like to get the improvements back into the standard library so everyone can benefit.
The improvements are:

- Raise the default recursion limit 
- Control of the path baked into the *.pyc file (`-s` to strip a prefix; `-p` to prepend a new one) to support different build/install locations.
- Compiling for multiple optimization levels at once (by specifying -o multiple times)

A PR should follow shortly.
msg353266 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-09-26 06:28 
New changeset 8e7bb991de7c88583bc6663d8bbc541054ca8dc4 by Petr Viktorin (Lumír 'Frenzy' Balhar) in branch 'master':
bpo-38112: Compileall improvements (GH-16012)
https://github.com/python/cpython/commit/8e7bb991de7c88583bc6663d8bbc541054ca8dc4
msg353271 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-26 07:12 
test_compileall failed on AMD64 Windows8.1 Non-Debug 3.x:
https://buildbot.python.org/all/#/builders/12/builds/3257

Example:

ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\test\test_compileall.py", line 48, in setUp
    os.makedirs(self.long_path)
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  [Previous line repeated 18 more times]
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 221, in makedirs
    mkdir(name, mode)
FileNotFoundError: [WinError 206] The filename or extension is too long: 'D:\\temp\\tmpcgsk1z3q\\long\\1\\2\\3\\4\\5\\6\\7\\8\\9\\10\\11\\12\\13\\14\\15\\16\\17\\18\\19\\20\\21\\22\\23\\24\\25\\26\\27\\28\\29\\30\\31\\32\\33\\34\\35\\36\\37\\38\\39\\40\\41\\42\\43\\44\\45\\46\\47\\48\\49\\50\\51\\52\\53\\54\\55\\56\\57\\58\\59\\60\\61\\62\\63\\64\\65\\66\\67\\68\\69\\70\\71\\72\\73\\74\\75\\76\\77\\78'

Failures:

ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_files (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_path (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_magic_number (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_mtime (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_optimize (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_strip_only (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_files (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_path (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_magic_number (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_mtime (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_optimize (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_strip_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)
FAIL: test_no_args_respects_force_flag (test.test_compileall.CommandLineTestsWithSourceEpoch)
Re-running failed tests in verbose mode
Re-running test_tarfile in verbose mode
Re-running test_compileall in verbose mode
ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_files (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_path (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_magic_number (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_mtime (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_optimize (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_strip_only (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_files (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_path (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_magic_number (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_mtime (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_optimize (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_strip_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)

pythoninfo says:

sys.windowsversion: sys.getwindowsversion(major=6, minor=3, build=9600, platform=2, service_pack='')
windows.RtlAreLongPathsEnabled: <function not available>
msg353272 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-26 07:16 
Similar errors on AMD64 Windows10 3.x:
https://buildbot.python.org/all/#/builders/3/builds/3567

pythoninfo:

sys.windowsversion: sys.getwindowsversion(major=10, minor=0, build=17134, platform=2, service_pack='')
windows.RtlAreLongPathsEnabled: False

--

And AMD64 Windows7 SP1 3.x:
https://buildbot.python.org/all/#/builders/40/builds/3180

pythoninfo:

sys.windowsversion: sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1')
windows.RtlAreLongPathsEnabled: <function not available>
msg353275 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-09-26 07:24 
Thanks!
The buildbot complained in the pull request a while ago; I commented there that I'm looking into it.

I'd appreciate a review for the fix; I don't have a good way to test it other than offer it to the buildbots.
msg353286 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-09-26 09:54 
New changeset 4267c989e7fc6cd528e8a1d04a07fac5cca85ec7 by Petr Viktorin in branch 'master':
bpo-38112: compileall: Skip long path path on Windows if the path can't be created (GH-16414)
https://github.com/python/cpython/commit/4267c989e7fc6cd528e8a1d04a07fac5cca85ec7
msg353319 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-26 15:16 
FYI test_compileall.test_compile_dir_maxlevels() also fails on AMD64 FreeBSD CURRENT Shared 3.x:
https://buildbot.python.org/all/#builders/168/builds/1529
msg354610 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-10-14 00:46 
The AMD64 Windows7 SP1 3.x buildbot has been failing for a while since commit 4267c989e7fc6cd528e8a1d04a07fac5cca85ec7, please take a look at https://bugs.python.org/issue38470.
msg354728 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-15 13:40 
Should be fixed by:

New changeset eb1dda2b56f67f09352c303588c28880c471ae87 by Petr Viktorin (Victor Stinner) in branch 'master':
bpo-38470: Fix test_compileall.test_compile_dir_maxlevels() (GH-16789)
https://github.com/python/cpython/commit/eb1dda2b56f67f09352c303588c28880c471ae87
msg362783 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-02-27 07:33 
https://bugs.python.org/issue39769 may be of interest.  compileall's ddir= does not behave properly in recursive subdirectories, it did the right thing in 2.7 which is similar to a combination of compile_dir(dir, prependdir=ddir, stripdir=dir) after this issue's feature.
msg369215 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-18 13:23 
New changeset adc72bb2f9a5d8b548ee04405e19a184e5699e8d by Lumír 'Frenzy' Balhar in branch 'master':
bpo-38112: Document that compileall.compile_[dir,file] also accept multiple opt levels (GH-20174)
https://github.com/python/cpython/commit/adc72bb2f9a5d8b548ee04405e19a184e5699e8d
History
Date User Action Args
2022-04-11 14:59:20adminsetgithub: 82293
2020-05-18 13:23:44vstinnersetmessages: + msg369215
2020-05-18 13:23:20frenzysetnosy: + frenzy

pull_requests: + pull_request19475
2020-02-27 07:33:08gregory.p.smithsetnosy: + gregory.p.smith
messages: + msg362783
2019-10-15 13:40:03vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg354728

stage: patch review -> resolved
2019-10-14 00:46:40pablogsalsetnosy: + pablogsal
messages: + msg354610
2019-09-26 15:16:00vstinnersetmessages: + msg353319
2019-09-26 11:56:29petr.viktorinsetpull_requests: + pull_request15999
2019-09-26 09:54:05petr.viktorinsetmessages: + msg353286
2019-09-26 07:24:44petr.viktorinsetmessages: + msg353275
2019-09-26 07:19:55petr.viktorinsetpull_requests: + pull_request15994
2019-09-26 07:16:23vstinnersetmessages: + msg353272
2019-09-26 07:12:58vstinnersetnosy: + vstinner
messages: + msg353271
2019-09-26 06:28:43petr.viktorinsetmessages: + msg353266
2019-09-12 04:54:12frenzysetkeywords: + patch
stage: patch review
pull_requests: + pull_request15638
2019-09-11 14:01:19petr.viktorincreate