Title: distutils doesn't byte-compile .py files to __pycache__ during installation
Type: behavior Stage: resolved
Components: Distutils, Distutils2 Versions: Python 3.2, Python 3.3, 3rd party
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: alexis, barry, dmalcolm, eric.araujo, georg.brandl, jramnani, ned.deily, python-dev, scoder, tarek
Priority: critical Keywords: patch

Created on 2011-02-20 09:54 by scoder, last changed 2012-02-11 04:47 by eric.araujo. This issue is now closed.

File name Uploaded Description Edit
issue11254.patch scoder, 2011-02-20 10:21 proposed patch, hopefully minimally invasive review
issue11254.patch jramnani, 2011-03-14 21:49 Update #1 to patch. Add fix for failing unit tests review
bug-11254.diff eric.araujo, 2011-03-20 21:32 review
Messages (19)
msg128897 - (view) Author: Stefan Behnel (scoder) * Date: 2011-02-20 09:54
During installation of Python packages ( install or bdist), distutils puts .pyc files into the installed source directory, instead of moving them into __pycache__. This may mean that they are not getting used after installation (with potentially no way of getting updated due to lack of write access by users), and that source files that get imported during installation may end up with .pyc files in both the source directory and the __pycache__ directory in the installed package.

The relevant python-dev thread is here:
msg128899 - (view) Author: Stefan Behnel (scoder) * Date: 2011-02-20 10:21
Here's a patch. I basically copied over the way py_compile determines the .pyc file name.

It works for me for a "normal" installation. However, I couldn't test it with "-O", as 2to3 crashes for me when I enable it during installation. I guess that's a separate issue.
msg128903 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-02-20 12:47
Patch looks good.  Regarding the problem with 2to3 and -O, maybe you can run 2to3 manually, copy the and run python3.2 -0.  disutils.util.byte_compile is not tested, so this patch requires at least careful manual testing, and if possible a unit test.

In distutils2, it may be possible to replace this function with a compileall call.
msg130912 - (view) Author: Jeff Ramnani (jramnani) * Date: 2011-03-14 21:49
This patch applies cleanly as of rev 89af3880ca57 on branch default.

After applying the patch, I had two unit test failures when running,
"/python.exe -m test -v test_distutils", on OS X 10.6.  The tests were failing because they were not looking in the __pycache__ directory for byte-compiled or optimized files.

Attaching an updated patch that fixes the unit tests that were breaking.
msg130963 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-03-15 11:21
Thanks for the test.  The patch is minimal and makes distutils work correctly with the new pyc handling in 3.2+, so I will apply it in a few days unless Tarek disagrees.
msg131546 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-03-20 21:32
I cleaned up the patch a bit.  In particular, I removed the dance around imp.cache_from_source that seemed unnecessary.  I tested with regular Python, with PYTHONOPTIMIZE and with PYTHONDONTWRITEBYTECODE.  Can you review and test too?
msg131837 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-03-23 01:57
The dance around imp.cache_from_source looks necessary, judging by Georg’s
msg132460 - (view) Author: Jeff Ramnani (jramnani) * Date: 2011-03-29 03:31
I've reviewed your patch and it looks good.  I appreciate the review and cleanup.

The tests succeed for me after applying your patch.  I also tested with PYTHONOPTIMIZE and PYTHONDONTWRITEBYTECODE and got the output I expected.
msg137212 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-05-29 18:49
Should this be a release blocker for 3.2.1?
msg137253 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2011-05-30 10:14
to be backported in packaging -- in a way that will make it work with previous python versions for the incoming 2.x backport
msg140448 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2011-07-15 18:01
FWIW, I've filed a bug about this issue for Fedora 15's python3 package here:

Looks like this has led to an extra .pyc in the old location for every Python 3 .py file in Fedora 15's various python3-* rpm packages (as generated by distutils).  

rpm's post-processing has been generating .pyc files in the (correct) __pycache__ location (via rpmbuild's brp-python-bytecompile script, which uses "compileall").

For examples, see that Fedora bug report; so far it's affected every built rpm I've looked at.

I believe the impact of the extra .pyc files is merely wasted disk space within our rpm-packaged Python 3.2 stack.

I'm thinking of applying the patch to our downstream python3.src.rpm
msg145098 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-07 16:21
I will commit this today or tomorrow.
msg145220 - (view) Author: Roundup Robot (python-dev) Date: 2011-10-09 06:59
New changeset 27a36b05caed by Éric Araujo in branch '3.2':
Fix distutils byte-compilation to comply with PEP 3147 (#11254).

New changeset 651e84363001 by Éric Araujo in branch '3.2':
Fix docstring of distutils.util.byte_compile (followup for #11254)

New changeset 439f47fadffa by Éric Araujo in branch 'default':
Fix packaging byte-compilation to comply with PEP 3147 (#11254).
msg145228 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-09 07:19
Finally fixed.  I fail at boolean logic; removing the “dance around imp.cache_from_source that seemed unnecessary” as I called it was a mistake, we have to do it, so I restored that part of your original patch.  I’m sorry I took so long to fix this, it was straightforward.
msg145942 - (view) Author: Roundup Robot (python-dev) Date: 2011-10-19 19:32
New changeset 1cc4d822123b by Éric Araujo in branch 'default':
More fixes for PEP 3147 compliance in packaging (#11254)
msg145943 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-19 19:38
Installation was not completely fixed.  Distutils delegates to each command’s get_outputs method to make the list of all files to install or package in an sdist, and the previous patch missed edits to these get_outputs methods.  I should have fixed it in packaging, I’ll port to distutils soon.
msg146939 - (view) Author: Roundup Robot (python-dev) Date: 2011-11-03 15:45
New changeset ea926dff958f by Éric Araujo in branch '3.2':
More fixes for PEP 3147 compliance in distutils (#11254)

New changeset 60ede940089f by Éric Araujo in branch 'default':
Merge follow-up for #11254 and other changes from 3.2
msg147681 - (view) Author: Roundup Robot (python-dev) Date: 2011-11-15 15:14
New changeset c10946a17420 by Éric Araujo in branch 'default':
Clean up byte-compilation code in packaging (#11254 followup).
msg153091 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-02-11 04:47
See also #13123 for the same bug in bdist_wininst.
Date User Action Args
2012-02-11 04:47:06eric.araujosetmessages: + msg153091
2011-11-15 15:14:00python-devsetmessages: + msg147681
2011-11-03 16:07:04eric.araujosetstatus: open -> closed
resolution: fixed
stage: commit review -> resolved
2011-11-03 15:45:54python-devsetmessages: + msg146939
2011-11-03 01:52:08pitrouunlinkissue13307 dependencies
2011-11-02 16:54:11eric.araujolinkissue13307 dependencies
2011-10-21 13:27:14eric.araujolinkissue13181 superseder
2011-10-19 19:38:18eric.araujosetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg145943

stage: resolved -> commit review
2011-10-19 19:32:55python-devsetmessages: + msg145942
2011-10-09 07:19:54eric.araujosetstatus: open -> closed
resolution: fixed
messages: + msg145228

stage: commit review -> resolved
2011-10-09 06:59:49python-devsetnosy: + python-dev
messages: + msg145220
2011-10-07 16:21:52eric.araujosetmessages: + msg145098
2011-07-15 18:01:24dmalcolmsetnosy: + dmalcolm
messages: + msg140448
2011-05-30 10:14:40tareksetmessages: + msg137253
2011-05-29 18:49:29ned.deilysetnosy: + ned.deily
messages: + msg137212
2011-03-29 03:31:25jramnanisetmessages: + msg132460
2011-03-23 01:57:57eric.araujosetnosy: + georg.brandl
messages: + msg131837
2011-03-20 21:32:06eric.araujosetfiles: + bug-11254.diff
nosy: barry, scoder, tarek, eric.araujo, alexis, jramnani
messages: + msg131546
2011-03-15 11:21:40eric.araujosetassignee: tarek -> eric.araujo
messages: + msg130963
nosy: barry, scoder, tarek, eric.araujo, alexis, jramnani
stage: patch review -> commit review
2011-03-14 21:49:22jramnanisetfiles: + issue11254.patch
nosy: + jramnani
messages: + msg130912

2011-02-20 14:00:36pitrousetpriority: normal -> critical
nosy: barry, scoder, tarek, eric.araujo, alexis
2011-02-20 12:47:22eric.araujosetversions: + 3rd party, Python 3.3
nosy: + alexis, barry

messages: + msg128903

components: + Distutils2
stage: patch review
2011-02-20 10:21:24scodersetfiles: + issue11254.patch

messages: + msg128899
keywords: + patch
nosy: scoder, tarek, eric.araujo
2011-02-20 09:54:53scodercreate