classification
Title: packaging: build command should have options to control byte-compilation
Type: Stage: resolved
Components: Distutils2 Versions: Python 3.3
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: Arfrever, Julien.Courteau, alexis, eric.araujo, tarek
Priority: normal Keywords: easy

Created on 2011-11-14 14:40 by Arfrever, last changed 2014-03-13 07:07 by eric.araujo. This issue is now closed.

Messages (16)
msg147600 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2011-11-14 14:40
build command of packaging should accept --compile, --no-compile and --optimize options and pass them to build_py command.
msg148319 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-11-25 15:15
Thinking about this, build_py --compile clearly refers to byte-compilation of Python modules, but the same option on the build command is more ambiguous: It could be interpreted to mean “skip C compilation”.
msg148333 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2011-11-25 16:37
Maybe --byte-compile and --no-byte-compile could be used.
msg148341 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-11-25 17:12
Juste a note: currently, when a command sets one option from the value given to another command, they have the same name, with two kinds of exceptions:
- build --build-lib becomes build_lib --build-dir (etc.)
- install --install-lib becomes install_lib --install-dir (etc.)

So, introducing --(no-)byte-compile and --optimize-bytecode= would change that scheme, but it is not a big deal.

On the third hand, maybe this is an opportunity to change the option names on the build_py and install* commands too.  For example, for a long time I thought that --optimize implied --compile, but actually they’re distinct options.

BTW, would these new options simplify some scripts you have, or do you just request them for consistency?
msg148344 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2011-11-25 17:59
Byte-compilation should be disabled during building of packages in Gentoo. PYTHONDONTWRITEBYTECODE="1" is set by default in environment. This variable affects distutils and until recently it affected packaging. --no-byte-compile will have to be explicitly passed to `pysetup${version} run build` and `pysetup${version} run install_dist` to disable byte-compilation.
msg148404 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-11-26 13:40
Okay, I think it’s a valuable use case.

(Out of curiosity, why don’t you want byte-compiled files on your system?  It speeds up imports, and problems due to the presence of stray pyc files when the py is deleted are gone in 3.2+.  Maybe you have custom tools to byte-compile, like Debian?)

Do you have any opinion about my renaming suggestion?
Without renaming, we’d have that:

  build --byte-compile --no-byte-compile --optimize-bytecode=[012]
  build_py --compile --no-compile --optimize=[012]

If we want to use the same name and make the names clearer, we could have:

  build(_py) --compile-pyc --no-compile-pyc --compile-pyo=[012]
msg148436 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2011-11-27 02:19
I suggest to not use "pyc" and "pyo" in options, because ".pyc" and ".pyo" filename extensions are specific to a subset of Python implementations. Jython uses "$py.class" filename extension (module$py.class for module.py).

You could use --byte-compile, --no-byte-compile and --optimize-bytecode (or --optimize-byte-code) for both build and build_py commands.

Do settings from setup.cfg affect byte-compilation?
msg148478 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-11-28 13:55
> I suggest to not use "pyc" and "pyo" in options, because ".pyc" and ".pyo" filename extensions
> are specific to a subset of Python implementations. Jython uses "$py.class" filename extension
> (module$py.class for module.py).
But these are extension modules, not Python modules byte-compiled for caching, are they?

> You could use --byte-compile, --no-byte-compile and --optimize-bytecode (or --optimize-byte-code)
> for both build and build_py commands.
I like the first two names, but still think that --optimize-bytecode might make people imply that the option is dependent on (or implying) --byte-compile.  Maybe it’s just me; I did not read the source for these options closely and was under this misconception for months.  OTOH, even with one maybe possibly ambiguous option name, I can still make the help text clearer (i.e. “byte-compile Python modules” and “byte-compile Python modules with optimizations”).

(We’re spending a lot of thoughts for a very marginally useful feature (.pyo files).)

> Do settings from setup.cfg affect byte-compilation?
Yes.  Like any command options, --(no-)compile and --optimize can be given in config files or on the command-line.

BTW, why don’t you want byte-compiled files on your system?
msg148499 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2011-11-28 16:15
Jython's *$py.class files are byte-compiled modules, not extension modules.

There should be a way to disable generation of *.pyo files on command line even if setup.cfg enables it.

IMHO it would make more sense if --optimize-bytecode was dependant on --byte-compile option:
--no-byte-compile                      -> No *.pyc and *.pyo
--byte-compile --optimize-bytecode=0   -> Only *.pyc
--byte-compile --optimize-bytecode=0,1 -> *.pyc and *.pyo (with docstrings)
--byte-compile --optimize-bytecode=0,2 -> *.pyc and *.pyo (without docstrings)
--byte-compile --optimize-bytecode=1   -> Only *.pyo (with docstrings)
--byte-compile --optimize-bytecode=2   -> Only *.pyo (without docstrings)
--byte-compile --optimize-bytecode=1,2 -> Error

Byte-compiled files in Gentoo are generated separately, after installation.
msg148501 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-11-28 16:24
> Jython's *$py.class files are byte-compiled modules, not extension modules.
Thanks for the data point.  Agreed distutils[2] should not say “pyc” and “pyo” then.

> There should be a way to disable generation of *.pyo files on command line even if
> setup.cfg enables it.
There is.  The precedence of options is: stdlib distutils.cfg < (overriden by) user .pydistutils.cfg < local setup.cfg < options on the command line.  Pass --no-compile --optimize=0 to never ever byte-compile (or pass --no-user-cfg and rely on the defaults).

> IMHO it would make more sense if --optimize-bytecode was dependant on --byte-compile option:
It was also my expectation, as I told.  The scheme that you propose keeps all current possibilities, it’s nice!  What do you think about conflating two options into one?

> --no-byte-compile         -> No *.pyc or *.pyo
> --byte-compile            -> Only *.pyc
> --byte-compile=0          -> Only *.pyc
> --byte-compile=0,1        -> *.pyc and *.pyo (level 1)
(etc.)
There may be a technical hurdle to overcome (not sure the option parsing system will allow 0 or more arg), but I’m asking for human interface feedback first.  (I’m just trying to make the list of options a bit smaller to reduce the overload, but if it feels complicated I won’t do it.)

> Byte-compiled files in Gentoo are generated separately, after installation.
Are you using standard py_compile or compileall modules or your own scripts?  I’ve seen that Debian for example has its own scripts and I’m sad to see no feature requests upstreamed to us instead.
msg148503 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2011-11-28 16:36
--byte-compile=arguments is a good idea.

(Gentoo uses py_compile and compileall modules.)
msg148504 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-11-28 16:46
Thanks for all the feedback!  So, for anyone interested in contributing, two patches are needed:

One that changes the existing options to use the new names and parsing behavior (--no-compile, --compile[=0,1,2]) and adds tests for the erroneous --compile=1,2 (other combinations are already covered).

A second one to add --no-compile and --compile to build, make build_py take its default values from build, and add tests to see if this works.

As usual, I’m available for help here, on the core-mentorship list or in private email.  (I don’t plan to do this myself in the short term because it’s a minor feature, I have an alpha to release and I have to keep the easiest things for new contributors.)
msg160504 - (view) Author: Julien Courteau (Julien.Courteau) * Date: 2012-05-13 02:55
Here is the last proposition of Eric done at the last Montreal Sprint (May 12):

--no-byte-compile    -> No *.pyc and *.pyo
--byte-compile=b     -> Only *.pyc
--byte-compile=b,o   -> *.pyc and *.pyo (with docstrings)
--byte-compile=b,oo  -> *.pyc and *.pyo (without docstrings)
--byte-compile=o     -> Only *.pyo (with docstrings)
--byte-compile=oo    -> Only *.pyo (without docstrings)
--byte-compile=o,oo  -> Error

The parameters are symetric with the -B, -o and -oo of the python command.
msg160670 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-05-14 22:12
FTR I asked Julien to provide his opinion as a user and he agreed that having just one --byte-compile/--no-byte-compile option pair seemed better than the existing names (where --compile and --optimize are unrelated).

I wanted to avoid --byte-compile=0 to generate pyc, because I expected people to interpret it as “disabled” instead of “optimization level 0”.  Thus the idea of using letters (and avoiding to use c and o as not all VMs have pyc and pyo files).

Note that the new spelling for --compile is the seemingly redundant --byte-compile=b, but that does not matter as it will be the default value so people won’t type it.
msg161372 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-05-22 19:05
One thing just came to my mind: if we change option names and people use the new names in setup.cfg or ~/.pydistutils.cfg, then the file will stop being compatible with distutils.
msg161472 - (view) Author: Julien Courteau (Julien.Courteau) * Date: 2012-05-24 01:33
It is possible to only change the "frontend" (options: byte-compile, compile, no-byte-compile, optimize) without changing the "backend" (attributes: compile and optimize). This way it is then easy to handle both sets of options (no-byte-compile, byte-compile, compile and optimize) for as long as it is necessary. I wonder if that could cause any problem...
History
Date User Action Args
2014-03-13 07:07:17eric.araujosetstatus: open -> closed
dependencies: - Add tests for files byte-compiled by distutils[2]
resolution: out of date
stage: resolved
2012-05-24 01:33:56Julien.Courteausetmessages: + msg161472
2012-05-22 19:05:22eric.araujosetmessages: + msg161372
2012-05-14 22:12:15eric.araujosetmessages: + msg160670
2012-05-13 02:55:20Julien.Courteausetnosy: + Julien.Courteau
messages: + msg160504
2011-11-28 16:46:31eric.araujosetdependencies: + Add tests for files byte-compiled by distutils[2]
messages: + msg148504
2011-11-28 16:36:21Arfreversetmessages: + msg148503
2011-11-28 16:24:55eric.araujosetassignee: tarek -> eric.araujo
messages: + msg148501
2011-11-28 16:15:09Arfreversetmessages: + msg148499
2011-11-28 13:55:59eric.araujosetmessages: + msg148478
2011-11-27 02:19:34Arfreversetmessages: + msg148436
2011-11-26 13:40:17eric.araujosetmessages: + msg148404
title: packaging: build command should accept --compile, --no-compile and --optimize options -> packaging: build command should have options to control byte-compilation
2011-11-25 17:59:10Arfreversetmessages: + msg148344
2011-11-25 17:12:03eric.araujosetmessages: + msg148341
2011-11-25 16:37:54Arfreversetmessages: + msg148333
2011-11-25 15:15:10eric.araujosetmessages: + msg148319
2011-11-14 14:40:22Arfrevercreate