classification
Title: py_compile: foo.bar.py → __pycache__/foo.cpython-34.pyc
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.5, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: barry Nosy List: Arfrever, barry, brett.cannon, doko, piotr, python-dev, r.david.murray
Priority: normal Keywords: 3.3regression, 3.4regression

Created on 2014-11-29 10:16 by piotr, last changed 2014-12-02 17:12 by barry. This issue is now closed.

Files
File name Uploaded Description Edit
22966.txt barry, 2014-12-01 21:59 review
Messages (10)
msg231859 - (view) Author: Piotr Ożarowski (piotr) Date: 2014-11-29 10:16
when py_compile module is asked to create .pyc file for file with invalid name, it doesn't fail or ignore such request - it creates/overwrites .pyc file for sane file name (i.e. ignores everything after dot)

$ touch foo.bar.py
$ python3.4 -m py_compile foo.bar.py
$ ls __pycache__
foo.cpython-34.pyc

Even though foo.bar.py is not a valid file name, some programmers are using such names for plugins which leads to bugs similar to https://lists.debian.org/debian-python/2014/11/msg00061.html.

I will update my scripts to not feed py_compile module with file names containing dot (and remove already generated .pyc files if such files are detected), but please do not generate them or generate .pyc files with invalid file names as well.
msg231866 - (view) Author: Piotr Ożarowski (piotr) Date: 2014-11-29 18:02
Python 3.2 generates foo.bar.cpython-32.pyc so it's a regression
msg231905 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-11-30 21:03
And, therefore, a missing test :(.  This probably snuck in when we refactored the CLI.
msg231940 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2014-12-01 15:30
seen as well with 3.3
msg231944 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2014-12-01 16:06
proposed patch:

diff -r 8dacb1a21793 Lib/importlib/_bootstrap.py
--- a/Lib/importlib/_bootstrap.py	Fri Nov 28 13:15:41 2014 +0100
+++ b/Lib/importlib/_bootstrap.py	Mon Dec 01 17:05:00 2014 +0100
@@ -453,7 +453,7 @@
     else:
         suffixes = OPTIMIZED_BYTECODE_SUFFIXES
     head, tail = _path_split(path)
-    base_filename, sep, _ = tail.partition('.')
+    base_filename, sep, _ = tail.rpartition('.')
     tag = sys.implementation.cache_tag
     if tag is None:
         raise NotImplementedError('sys.implementation.cache_tag is None')
msg231946 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2014-12-01 16:31
I'll take this one.  I think it should be easy to add a test case, which I'll do.
msg231956 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2014-12-01 21:59
Not counting importlib.h, here's the diff I'm going to apply to 3.4.  It passes all the existing tests and includes a new test for this behavior.
msg231963 - (view) Author: Roundup Robot (python-dev) Date: 2014-12-01 23:17
New changeset 269bf37a57a1 by Barry Warsaw in branch '3.4':
- Issue #22966: Fix __pycache__ pyc file name clobber when pyc_compile is
https://hg.python.org/cpython/rev/269bf37a57a1

New changeset 3b3ba38d503a by Barry Warsaw in branch '3.4':
- Issue #22966: Fix __pycache__ pyc file name clobber when pyc_compile is
https://hg.python.org/cpython/rev/3b3ba38d503a

New changeset 25113281d543 by Barry Warsaw in branch 'default':
- Issue #22966: Fix __pycache__ pyc file name clobber when pyc_compile is
https://hg.python.org/cpython/rev/25113281d543
msg232010 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-12-02 16:08
Apparently this broke under Windows: http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/8999/steps/test/logs/stdio
msg232027 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2014-12-02 17:12
I already pushed a fix.

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.4/builds/702/steps/test/logs/stdio

(although asyncio is still failing there but that should be unrelated)
History
Date User Action Args
2014-12-02 17:12:20barrysetstatus: open -> closed

messages: + msg232027
2014-12-02 16:57:36Arfreversetnosy: + Arfrever
2014-12-02 16:08:59brett.cannonsetstatus: closed -> open

messages: + msg232010
2014-12-02 02:06:46berker.peksagsetstage: patch review -> resolved
2014-12-01 23:19:48barrysetstatus: open -> closed
resolution: fixed
2014-12-01 23:17:54python-devsetnosy: + python-dev
messages: + msg231963
2014-12-01 21:59:18barrysetfiles: + 22966.txt

messages: + msg231956
2014-12-01 16:32:50barrysetversions: + Python 3.5
2014-12-01 16:31:41barrysetassignee: barry
messages: + msg231946
2014-12-01 16:06:05dokosetmessages: + msg231944
stage: test needed -> patch review
2014-12-01 15:30:26dokosetkeywords: + 3.3regression, 3.4regression
nosy: + doko
messages: + msg231940

2014-11-30 21:03:41r.david.murraysetnosy: + r.david.murray
messages: + msg231905
2014-11-29 18:02:01piotrsetmessages: + msg231866
2014-11-29 17:22:30barrysetnosy: + barry
2014-11-29 14:33:31brett.cannonsetnosy: + brett.cannon
2014-11-29 14:33:15brett.cannonsettype: behavior
components: + Library (Lib)
stage: test needed
2014-11-29 10:16:42piotrcreate