classification
Title: Add tests for files byte-compiled by distutils
Type: Stage: needs patch
Components: Distutils Versions: Python 3.5, Python 3.4, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, alexis, dstufft, eric.araujo, ncoghlan, r.david.murray, tarek
Priority: normal Keywords: patch

Created on 2011-11-24 14:45 by eric.araujo, last changed 2014-10-04 23:29 by r.david.murray.

Files
File name Uploaded Description Edit
p7g-tests-bytecompiled-files.diff eric.araujo, 2011-11-24 14:45 review
Messages (7)
msg148258 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-11-24 14:45
I recently changed packaging.util.byte_compile, the function used by the build_py and install_lib commands, so that it can create .pyc and/or .pyo files independently of the calling’s Python -O or -B flags (dad02a080bbc), but I think I introduced a bug.  To check that bug and fully test byte_compile before I further simplify the function and backport the whole fix to distutils, I wanted to add tests that load a .pyc or .pyo file and make sure that the right optimization level is used (nothing for --compile, level one for --optimize=1, level 2 for --optimize=2).

In the attached patch, I run a Python with -O, -OO or no flag in a subprocess to import a module that prints different things depending on the optimization level.  The .pyc or .pyo file is already created by packaging, but I have no check that makes sure that they are not recreated (which I don’t want) and I don’t know if the optimizations in compile.c are applied during byte-compilation or after import (which would make my tests not exercise what I want).

(I tried to use imp.load_module at first but it’s not very friendly.  Other ideas that I had after the patch was done: Use subprocess + imp so that I can import pyc or pyo as I wish; use open + dis module.)

So, could you import experts review the functions added in packaging.tests.support and tell me if they exercise what I want?
msg148260 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-11-24 14:57
(For the reference, the bug I added is this: http://hg.python.org/cpython/rev/c10946a17420#l6.45

p7g.util.byte_compile calls py_compile.compile with a filename ending in .pyc or .pyo as appropriate, but the optimization level in the py_compile.compile function depends on the calling Python’s -O option.  When byte_compile wants to create a .pyo file but is run under a python without -O, it spawns a subprocess with the -O option.  The fix I did made sure that the filename used (pyc or pyo) did not depend on the calling Python’s -O, *but* it should not have removed the setting of -O/-OO in the subprocess, as it’s the only way to control the optimization level in py_compile.compile before 3.2.  I think I will use a py_compile or compileall function so that I can have control over the optimization level from Python, which will let me remove the inelegant spawning in byte_compile, and I will backport that function for distutils2.

If this makes no sense, please ignore.  I’m not sure I would have understood what I’m talking about a year ago.)
msg148437 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2011-11-27 02:26
What exactly do you consider to backport to distutils?
msg148477 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-11-28 13:49
See http://bugs.python.org/issue12119#msg146943
msg151178 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-01-13 15:59
Nick, would you have a bit of time to read my OP and reply?
msg159349 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-04-26 01:38
Your basic approach looks sensible to me.

One trick I use in test_cmd_line_script to prevent recreation is to simply delete the source file. If the source is gone, implicit recreation is impossible. Unfortunately, that doesn't work for __pycache__, since the cached version will be ignored if the original goes missing.

One useful explicit check (as per #14443) would be to ensure the magic number and other attributes are as expected:
http://hg.python.org/cpython/file/57d558f1904d/Lib/importlib/_bootstrap.py#l444

You can also pass the "-B" flag to your testing subprocesses, which will switch off the implicit bytecode generation.
msg228510 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-10-04 23:29
Looks like someone should revise this patch taking in to account Nick's feedback, and updating it to apply to distutils instead of the now-defunct distutils2/packaging.  

Moving stage to 'needs patch'.
History
Date User Action Args
2014-10-04 23:29:46r.david.murraysetassignee: eric.araujo ->

components: - Distutils2
title: Add tests for files byte-compiled by distutils[2] -> Add tests for files byte-compiled by distutils
nosy: + dstufft, r.david.murray
versions: + Python 3.4, Python 3.5, - 3rd party, Python 3.2, Python 3.3
messages: + msg228510
stage: commit review -> needs patch
2014-03-13 07:07:17eric.araujounlinkissue13400 dependencies
2013-02-01 22:30:38brett.cannonsetnosy: - brett.cannon
2012-04-26 01:38:16ncoghlansetmessages: + msg159349
2012-01-13 15:59:08eric.araujosetmessages: + msg151178
2011-11-28 16:46:31eric.araujolinkissue13400 dependencies
2011-11-28 13:49:19eric.araujosetmessages: + msg148477
2011-11-27 02:26:02Arfreversetnosy: + Arfrever
messages: + msg148437
2011-11-24 14:57:40eric.araujosetmessages: + msg148260
2011-11-24 14:45:11eric.araujocreate