This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: logger.StreamHandler emit encoding fallback is wrong
Type: behavior Stage:
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: vinay.sajip Nosy List: cboos, eric.araujo, vinay.sajip
Priority: normal Keywords: patch

Created on 2009-12-10 13:14 by cboos, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
logging_error.py cboos, 2009-12-10 13:14 script for reproducing the issue, for Python 2.7a1 will trigger an UnicodeDecodeError: 'ascii' codec can't decode ...
issue7470.patch cboos, 2009-12-10 13:18 Avoid spurious re-decoding of just UTF-8 encoded string, in the logger.StreamLogger.
Messages (7)
msg96201 - (view) Author: Christian Boos (cboos) * Date: 2009-12-10 13:14
For a stream with a "poor" encoding, such as sys.stderr on Windows where 
encoding is cp437, the encoding of an unicode message will fail 
(expected) and then a fallback encoding to UTF-8 will be done:

(in 
http://code.python.org/hg/trunk/file/bd98b2c097fe/Lib/logging/__init__.p
y#l837):
837                 except UnicodeError:
838                     stream.write(fs % msg.encode("UTF-8"))

However, that fallback won't work, as at this point, fs was already 
converted to unicode and is now u'%s\n', so the (msg.encode("UTF-8")) 
str will be decoded to unicode again using the default encoding, which 
will likely fail with a UnicodeDecodeError if msg contains non-ascii 
characters.

The solution would be to keep using fs as "%s\n" in this line.

This is similar to issue6991, but not exactly the same and it only 
happens for Python 2.7. Using logging_error.py, I've tested Python 2.3 
to Python 2.6 (works) and Python 2.7a1 (fails), current trunk must have 
the same issue as 2.7a1. Patch follows.
msg96202 - (view) Author: Christian Boos (cboos) * Date: 2009-12-10 13:18
One way to solve the issue. 

PS: copy/pasting URLs to the Mercurial repo apparently doesn't work, for 
some reason (probably the line anchor); the URL I gave above was:

 http://code.python.org/hg/trunk/file/bd98b2c097fe/Lib/logging/__init__.py

 + #l837
msg96203 - (view) Author: Christian Boos (cboos) * Date: 2009-12-10 13:21
PPS: Ok, links to the Mercurial repo /really/ don't work, and it was not 
the #l837 part ;-)
msg96231 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2009-12-11 00:52
Sorry Christian, your patch appears just to rename 'fs' to 'ufs' so I'm
not sure if your patch is exactly what you intended.
msg96238 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2009-12-11 09:17
Never mind my earlier comment, I looked at the change more carefully
after a cup of coffee, and it now makes sense to me ;-)

Fixed checked into trunk (r76754), thanks.
msg107644 - (view) Author: Christian Boos (cboos) * Date: 2010-06-12 11:46
Hello Vinay, 

I recently installed Python 2.6.5, and I see this buglet is also present there. When I reported the issue, I said 2.6 worked and IIRC it must have been 2.6 or 2.6.1. 

I don't know if it's worth to backport the fix for 2.6.6, but just in case, I wanted to let you know.
msg107645 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-06-12 11:50
Only security fixes and documentation fixes are allowed to go into stable branches.
History
Date User Action Args
2022-04-11 14:56:55adminsetgithub: 51719
2010-06-12 11:50:17eric.araujosetnosy: + eric.araujo
messages: + msg107645
2010-06-12 11:46:48cboossetmessages: + msg107644
2009-12-11 09:17:42vinay.sajipsetstatus: open -> closed
resolution: fixed
messages: + msg96238
2009-12-11 00:52:01vinay.sajipsetassignee: vinay.sajip
messages: + msg96231
2009-12-10 13:21:51cboossetmessages: + msg96203
2009-12-10 13:18:57cboossetfiles: + issue7470.patch
keywords: + patch
messages: + msg96202
2009-12-10 13:14:34cbooscreate