classification
Title: py_compile.compile raises if a file has bad encoding
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.4, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: berker.peksag Nosy List: Arfrever, berker.peksag, bkabrda, python-dev, vstinner
Priority: normal Keywords: patch

Created on 2013-10-30 09:42 by bkabrda, last changed 2014-09-01 09:30 by berker.peksag. This issue is now closed.

Files
File name Uploaded Description Edit
00186-dont-raise-from-py_compile.patch bkabrda, 2013-10-30 09:42
dont-raise-from-py_compile-test-included.patch bkabrda, 2013-10-30 10:24 review
Messages (10)
msg201721 - (view) Author: Bohuslav "Slavek" Kabrda (bkabrda) * Date: 2013-10-30 09:42
If py_compile.compile is used on a file with bad encoding (e.g. Lib/test/bad_coding2.py), the function raises even if doraise=False is passed. I'm attaching a patch that fixes this in 3.3 - I haven't tried on 3.4 yet and the code has changed, so I'm not sure it's problem there.

(Background: During RPM build of Python 3 in Fedora, we use py_compile.compile to make sure all files are properly compiled and have newer timestamps. We use 'find' to get all *.py files under 'python3.3/' and then xargs to pass the files to Python script that compiles them. If one of the files causes py_compile.compile to raise, the rest doesn't get compiled.)
msg201723 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-10-30 09:48
py_compile.compile() has been modified in Python 3.4. The encoding is now detected in the try/except block.

You should write a unit test for your patch.
http://docs.python.org/devguide/runtests.html#writing
msg201725 - (view) Author: Bohuslav "Slavek" Kabrda (bkabrda) * Date: 2013-10-30 10:24
Ok, I'm attaching a patch for 3.3 with a test case included.
msg225685 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-08-22 17:18
New changeset 38688f0a0e04 by Berker Peksag in branch '3.4':
Issue #19447: Add a test case to py_compile.compile() to make sure
http://hg.python.org/cpython/rev/38688f0a0e04

New changeset efb0a3eb4e06 by Berker Peksag in branch 'default':
Issue #19447: Add a test case to py_compile.compile() to make sure
http://hg.python.org/cpython/rev/efb0a3eb4e06
msg225686 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2014-08-22 17:23
3.3 is in security-only mode and this issue has been fixed in 3.4. I've just committed your test case to 3.4 and default. Thanks for the patch, Bohuslav.
msg225688 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2014-08-22 17:35
You should use importlib.util.cache_from_source(bad_coding) instead of bad_coding + 'c'.
msg225689 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-08-22 17:52
New changeset d99014320220 by Berker Peksag in branch '3.4':
Issue #19447: Use importlib.util.cache_from_source() instead of ``bad_coding + 'c'``.
http://hg.python.org/cpython/rev/d99014320220

New changeset 78fff5310206 by Berker Peksag in branch 'default':
Issue #19447: Use importlib.util.cache_from_source() instead of ``bad_coding + 'c'``.
http://hg.python.org/cpython/rev/78fff5310206
msg225690 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2014-08-22 17:53
Good point. Thank you Arfrever.
msg226171 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2014-08-31 11:41
I noticed this output of full test suite:

...
[361/388] test_bisect
[362/388] test_py_compile
  File "/tmp/cpython/Lib/test/bad_coding2.py", line 0
SyntaxError: encoding problem: utf8 with BOM

[363/388] test_netrc
[364/388] test_sys
...

I suggest that test_bad_coding() suppress output of py_compile.compile().
msg226223 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-09-01 09:29
New changeset 2d0bcb653085 by Berker Peksag in branch '3.4':
Issue #19447: Suppress output of py_compile.compile().
http://hg.python.org/cpython/rev/2d0bcb653085

New changeset a8ef9d7c4d20 by Berker Peksag in branch 'default':
Issue #19447: Suppress output of py_compile.compile().
http://hg.python.org/cpython/rev/a8ef9d7c4d20
History
Date User Action Args
2014-09-01 09:30:29berker.peksagsetstatus: open -> closed
resolution: fixed
stage: resolved
2014-09-01 09:29:32python-devsetmessages: + msg226223
2014-08-31 11:41:37Arfreversetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg226171

stage: resolved -> (no value)
2014-08-22 17:53:39berker.peksagsetmessages: + msg225690
2014-08-22 17:52:33python-devsetmessages: + msg225689
2014-08-22 17:35:26Arfreversetmessages: + msg225688
2014-08-22 17:23:35berker.peksagsetstatus: open -> closed

resolution: fixed

assignee: berker.peksag
stage: resolved
versions: + Python 3.4, Python 3.5, - Python 3.3
nosy: + berker.peksag
messages: + msg225686
components: + Library (Lib), - Build
type: behavior
2014-08-22 17:18:08python-devsetnosy: + python-dev
messages: + msg225685
2013-11-01 00:45:21Arfreversetnosy: + Arfrever
2013-10-30 10:24:45bkabrdasetfiles: + dont-raise-from-py_compile-test-included.patch

messages: + msg201725
2013-10-30 09:48:52vstinnersetnosy: + vstinner
messages: + msg201723
2013-10-30 09:42:07bkabrdacreate