When SyntaxError occurs and the module contains 
source encodings definition, current implementation 
prints error line in UTF8. This patch reverts the line into 
original encoding for printing.

This patch calls some memory-allocation APIs such as 
PyUnicode_DecodeUTF8. I'm not sure I can (or should) 
call PyErr_Clear() here if error happened.

Logged In: YES 
user_id=33168

I'm hoping that someone more familiar with unicode could
take a look at this.  The patch looks ok to me, but I
don't know how to test that it works.  I'm inclined to accept
it, unless I hear otherwise.

Logged In: YES 
user_id=38388

Please use the "replace" error handler when recoding the
source line
to Unicode - this will reduce the probability of the
conversion failing.

If you do get an error, it's likely going to be an unknown
encoding or
less likely a memory problem. Please add some logic to deal
with these
errors as well - currently you don't call PyError_Clear() or
take some
other action which may lead to confusing error reports (e.g.
error
popping up randomly during program execution due to the set
error).

Logged In: YES 
user_id=463672

Thanks for your comments. I'll post a revised patch and test
case later.

Logged In: YES 
user_id=463672

Sorry for my laziness. I revised a patch for current trunk.

- Use "replace" for recoding source
- Reports error with PyErr_Print()
- Test case

Logged In: YES 
user_id=33168

Note to self (or anyone interested): remember to look into this.

I think Martin von Loewis knows more about this.

Thanks for the patch. It wouldn't work as-is, because it broke PGEN. I
fixed that, and committed the change as r57961 and r57962.

We should make sure this is *not* merged into Py3k; there, things remain
unicode until they're printed, at which point the only encoding that
matters is the output file's encoding.

ishimoto: in dec_utf8, there is a PyErr_Print call. What is the purpose
of this call?

PyErr_Print() is called to report exception raised by codec. 
If PyUnicode_DecodeUTF8() or PyUnicode_AsEncodedString() return NULL,
PyErr_Print() is called.

> PyErr_Print() is called to report exception raised by codec.
> If PyUnicode_DecodeUTF8() or PyUnicode_AsEncodedString() return NULL,
> PyErr_Print() is called.

This comment is not very helpful; it describes what happens, but not
why, or whether that is a good idea. I believe that if this call is
ever reached, two tracebacks will be printed, confusing the user.

Sorry for insufficient comment.

When a codec raised an exception, I think the exception should be 
reported. Otherwise, user cannot know why Python prints broken line
of code.  

Should we silently clear the exception raised by codecs, or print a
message such as "Codec raised an exception while processing compile
error." ?

> Should we silently clear the exception raised by codecs, or print a
> message such as "Codec raised an exception while processing compile
> error." ?

Can you create a test case that triggers that specific problem?

Regards,
Martin

Codecs would hardly ever raises exception here. 
Usually, exception raised here would be a MemoryError. The unicode 
string we are trying to encode is just decoded by same codec. If codec
raises exception other than MemoryError, the codec will likely have problem.

I attached a script to print exception raised by codec. I wrote a "buggy" 
encoder, which never return string but raises an exception.

There are tons of situations where such an exception will be
suppressed, ofr better or for worse. I don't think this one deserves
such a radical approach.

On 10/11/07, atsuo ishimoto <report@bugs.python.org> wrote:
>
> atsuo ishimoto added the comment:
>
> Codecs would hardly ever raises exception here.
> Usually, exception raised here would be a MemoryError. The unicode
> string we are trying to encode is just decoded by same codec. If codec
> raises exception other than MemoryError, the codec will likely have problem.
>
> I attached a script to print exception raised by codec. I wrote a "buggy"
> encoder, which never return string but raises an exception.
>
> _____________________________________
> Tracker <report@bugs.python.org>
> <http://bugs.python.org/issue1031213>
> _____________________________________
>

That's fine with me. Please replace PyErr_Print() with PyErr_Clear().

> atsuo ishimoto added the comment:
>
> That's fine with me. Please replace PyErr_Print() with PyErr_Clear().

Done.

Committed revision 58471.

In release25-maint, PyErr_Print() should be replaced with PyErr_Clear()
also.

> In release25-maint, PyErr_Print() should be replaced with
> PyErr_Clear() also.

Committed revision 58991.