Title: UTF-8 incremental decoder doesn't support surrogatepass correctly
Type: behavior Stage: patch review
Components: Interpreter Core, Unicode Versions: Python 3.6, Python 3.5
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: RalfM, ezio.melotti, serhiy.storchaka, vstinner
Priority: high Keywords: patch

Created on 2015-05-16 22:56 by RalfM, last changed 2016-08-02 18:53 by serhiy.storchaka.

File name Uploaded Description Edit
Demo.txt RalfM, 2015-05-16 22:56 File to demonstrate the issue
surrogatepass.patch vstinner, 2016-07-27 16:33 review
Messages (4)
msg243376 - (view) Author: (RalfM) Date: 2015-05-16 22:56
I have an utf-8 encoded file containing single surrogates. Reading this file, specifying surrgatepass, works fine when I read the whole file with .read(), but raises an UnicodeDecodeError when I read the file line by line:

----- start of demo -----
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AM
D64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> with open("Demo.txt", encoding="utf-8", errors="surrogatepass") as f:
...   s =
>>> "\ud900" in s
>>> with open("Demo.txt", encoding="utf-8", errors="surrogatepass") as f:
...   for line in f:
...     pass
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "C:\Python\34x64\lib\", line 319, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 8190: inval
id continuation byte
----- end of demo -----

I attached the file used for the demo such that you can reproduce the problem.

If I change all 0xED bytes in the file to 0xEC (i.e. effectively change all surrogates to non-surrogates), the problem disappears.

The original file I noticed the problem with was 73 MB.  The demo file was derived from the original by removing data around the critical section, keeping the alignment to 16-k-blocks, and then replacing all printable ASCII characters by x.

If I change the file length by adding or removing 16 bytes to / from the beginning of the demo file, the problem disappears, so alignment seems to be an issue.

All this seems to indicate that the utf-8 decoder has problems when used for incremental decoding and a single surrogate appears around the block boundary.
msg271412 - (view) Author: (RalfM) Date: 2016-07-26 20:50
I just tested Python 3.6.0a3, and that (mis)behaves exactly like 3.4.3.
msg271461 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-07-27 16:33
Attached patch fixes the UTF-8 decoder to support correctly incremental decoder using surrogatepass error handler.

The bug occurs when b'\xed\xa4\x80' is decoded in two parts: the first two bytes (b'\xed\xa4'), and then the last byte (b'\x80').

It works as expected if we decode the first byte (b'\xed') and then the two last bytes (b'\xa4\x80').

My patch tries to keep best performances for the UTF-8/strict decoder.

@Serhiy: Would you mind to review my patch since you helped to design the fast UTF-8 decoder?
msg271839 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-08-02 18:53
The patch slows down decoding up to 20%.

$ ./python -m timeit -s 'b = b"\xc4\x80"*10000' -- 'b.decode()'
Unpatched:  10000 loops, best of 3: 50.8 usec per loop
Patched:    10000 loops, best of 3: 63.3 usec per loop

And I'm not sure that fixing only for the surrogatepass handler is enough. Other standard error handlers look working, but what if a user handler consumes more then one byte?
Date User Action Args
2016-08-02 18:53:51serhiy.storchakasetpriority: normal -> high

messages: + msg271839
components: + Interpreter Core
2016-07-27 17:41:02serhiy.storchakasetnosy: + serhiy.storchaka
stage: patch review

versions: + Python 3.5, - Python 3.4
2016-07-27 16:33:32vstinnersettitle: Exception with utf-8, surrogatepass and incremental decoding -> UTF-8 incremental decoder doesn't support surrogatepass correctly
2016-07-27 16:33:22vstinnersetfiles: + surrogatepass.patch
keywords: + patch
messages: + msg271461
2016-07-26 20:50:35RalfMsetmessages: + msg271412
versions: + Python 3.6
2015-05-16 22:56:34RalfMcreate