classification
Title: distutils: unable to compile both .opt-1.pyc and .opt2.pyc simultaneously
Type: enhancement Stage: resolved
Components: Distutils Versions: Python 3.6
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, cjmayo, dstufft, eric.araujo, mgorny, ned.deily, steve.dower
Priority: normal Keywords: patch

Created on 2016-06-05 06:03 by mgorny, last changed 2021-02-03 18:04 by steve.dower. This issue is now closed.

Files
File name Uploaded Description Edit
0001-distutils-make-OO-enable-both-opt-1-and-opt-2-optimi.patch mgorny, 2016-06-05 06:49 Patch making -OO compile both .opt-1 and .opt-2 review
Messages (9)
msg267389 - (view) Author: Michał Górny (mgorny) * Date: 2016-06-05 06:03
Since 3.5, optimized Python modules for -O and -OO are installed using different names. The Python build system itself seems to been prepared for this as Python itself is built with both .opt-1.pyc and .opt-2.pyc files built. However, distutils at the moment can only install one of them.

Gentoo is relying on 'setup.py install --compile -O2' doing the right thing. We can change the arguments if necessary but calling it twice would require a lot of conditional changes to how we build packages. Of course, not installing both variants is not an option since it causes Python to attempt to write them at runtime later.

Could we please get some way to build both .opt-1.pyc and .opt-2.pyc in setup.py install? Either through making -O2 build also -O1 implicitly, or providing an ability to specify multiple optimization levels.

Relevant Gentoo bug: https://bugs.gentoo.org/585060
msg267402 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2016-06-05 08:17
I would suggest using --optimize=1,2 / -O1,2 for this purpose.

Documentation in this part of Lib/distutils/command/build_py.py and Lib/distutils/command/install_lib.py should also be updated:

        ('optimize=', 'O',
         "also compile with optimization: -O1 for \"python -O\", "
         "-O2 for \"python -OO\", and -O0 to disable [default: -O0]"),
msg267435 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-06-05 17:08
Since this is an addition to the changes made for PEP 488 (Issue23731), nosying Brett
msg267450 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-06-05 18:26
So the reason Python does the right thing at install is because compileall is used by the Makefile to do the compilation and it's run twice (it actually doesn't use distutils for this).

My question is what did distutils do before PEP 488 since .pyc and .pyo files had different names before as well? Did distutils build both previously and so this is a regression? Or is this a feature request?
msg267451 - (view) Author: Michał Górny (mgorny) * Date: 2016-06-05 18:33
Brett, .pyc was controlled by --compile and .pyo by --optimize (either 1 or 2). Technically only two variants could be used simultaneously, and distutils accounted for that.

Now you have .pyc + .opt-1.pyc + .opt-2.pyc, so three variants instead of two.
msg267458 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-06-05 18:54
But distutils still does the right thing with -O2 and --optimize 2, right? If that's the case then this is an enhancement request to add support to distutils to compile all bytecode levels and not a regression.
msg267462 - (view) Author: Michał Górny (mgorny) * Date: 2016-06-05 19:11
Yes, you could put it like this.
msg379050 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2020-10-19 22:45
FWIW the behaviour in distutils was intended to be controlled by the setup.p options, but in fact also depended on the options of the python command itself (-O and dont-write-bytecode option/envvar).  This was changed in distutils2 but judged too big to fix in distutils: https://hg.python.org/distutils2/rev/7c0a88497b5c
msg386238 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-02-03 18:04
Distutils is now deprecated (see PEP 632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools
History
Date User Action Args
2021-02-03 18:04:27steve.dowersetstatus: open -> closed

nosy: + steve.dower
messages: + msg386238

resolution: out of date
stage: resolved
2020-11-06 19:36:33brett.cannonsetnosy: - brett.cannon
2020-10-19 22:45:55eric.araujosetmessages: + msg379050
2016-07-24 10:50:15cjmayosetnosy: + cjmayo
2016-06-05 19:12:36brett.cannonsetversions: - Python 3.5
2016-06-05 19:12:30brett.cannonsettype: behavior -> enhancement
2016-06-05 19:11:16mgornysetmessages: + msg267462
2016-06-05 18:54:10brett.cannonsetmessages: + msg267458
2016-06-05 18:33:05mgornysetmessages: + msg267451
2016-06-05 18:26:58brett.cannonsetmessages: + msg267450
2016-06-05 17:08:28ned.deilysetnosy: + ned.deily, brett.cannon
messages: + msg267435
2016-06-05 08:17:13Arfreversetnosy: + Arfrever

messages: + msg267402
versions: + Python 3.6
2016-06-05 06:49:27mgornysetfiles: + 0001-distutils-make-OO-enable-both-opt-1-and-opt-2-optimi.patch
keywords: + patch
2016-06-05 06:03:28mgornycreate