I have an utf-8 encoded file containing single surrogates. Reading this file, specifying surrgatepass, works fine when I read the whole file with .read(), but raises an UnicodeDecodeError when I read the file line by line:

----- start of demo -----
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AM
D64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> with open("Demo.txt", encoding="utf-8", errors="surrogatepass") as f:
...   s = f.read()
...
>>> "\ud900" in s
True
>>> with open("Demo.txt", encoding="utf-8", errors="surrogatepass") as f:
...   for line in f:
...     pass
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "C:\Python\34x64\lib\codecs.py", line 319, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 8190: inval
id continuation byte
>>>
----- end of demo -----

I attached the file used for the demo such that you can reproduce the problem.

If I change all 0xED bytes in the file to 0xEC (i.e. effectively change all surrogates to non-surrogates), the problem disappears.

The original file I noticed the problem with was 73 MB.  The demo file was derived from the original by removing data around the critical section, keeping the alignment to 16-k-blocks, and then replacing all printable ASCII characters by x.

If I change the file length by adding or removing 16 bytes to / from the beginning of the demo file, the problem disappears, so alignment seems to be an issue.

All this seems to indicate that the utf-8 decoder has problems when used for incremental decoding and a single surrogate appears around the block boundary.

I just tested Python 3.6.0a3, and that (mis)behaves exactly like 3.4.3.

Attached patch fixes the UTF-8 decoder to support correctly incremental decoder using surrogatepass error handler.

The bug occurs when b'\xed\xa4\x80' is decoded in two parts: the first two bytes (b'\xed\xa4'), and then the last byte (b'\x80').

It works as expected if we decode the first byte (b'\xed') and then the two last bytes (b'\xa4\x80').

My patch tries to keep best performances for the UTF-8/strict decoder.

@Serhiy: Would you mind to review my patch since you helped to design the fast UTF-8 decoder?

The patch slows down decoding up to 20%.

$ ./python -m timeit -s 'b = b"\xc4\x80"*10000' -- 'b.decode()'
Unpatched:  10000 loops, best of 3: 50.8 usec per loop
Patched:    10000 loops, best of 3: 63.3 usec per loop

And I'm not sure that fixing only for the surrogatepass handler is enough. Other standard error handlers look working, but what if a user handler consumes more then one byte?

PR 12603 fixes this issue in more general way and does not affect performance.

New changeset 7a465cb5ee7e298cae626ace1fc3e7d97df79f2e by Serhiy Storchaka in branch 'master':
bpo-24214: Fixed the UTF-8 incremental decoder. (GH-12603)
https://github.com/python/cpython/commit/7a465cb5ee7e298cae626ace1fc3e7d97df79f2e

New changeset bd48280cb66544827952ca91e326cbb178c8c461 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.7':
bpo-24214: Fixed the UTF-8 incremental decoder. (GH-12603) (GH-12627)
https://github.com/python/cpython/commit/bd48280cb66544827952ca91e326cbb178c8c461

This change seems to have caused test failure reported in https://github.com/python-hyper/wsproto/issues/126


from codecs import getincrementaldecoder
decoder = getincrementaldecoder("utf-8")()
print(decoder.decode(b'f\xf1\xf6rd', False))

# With this commit 7a465cb5ee

➜  cpython git:(7a465cb5ee) ./python.exe /tmp/foo.py
f

Before 7a465cb5ee

➜  cpython git:(38f4e468d4) ./python.exe /tmp/foo.py
Traceback (most recent call last):
  File "/tmp/foo.py", line 3, in <module>
    print(decoder.decode(b'f\xf1\xf6rd', False))
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 1: invalid continuation byte

> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 1: invalid continuation byte

Python is right: b'f\xf1\xf6rd' is not a valid UTF-8 string:

$ python3
Python 3.7.3 (default, May 11 2019, 00:38:04) 
>>> b'f\xf1\xf6rd'.decode()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 1: invalid continuation byte

This change is deliberate: it makes UTF-8 incremental decoder correct (respect the UTF-8 standard). I close the issue.

Victor, I think you misunderstood the issue. The problem is not that a decoding error is raised. The problem is that the incremental decoder no longer raises where it raised before.

I think that both behavior may be correct, and that it is better to not rely on ability of the incremental decoder with final=False to detect an invalid encoded data, but I see now that it is possible to fix for the original issue more carefully, without changing that behavior. PR 14304 does this.

It also change the UTF-16 incremental decoder with the surrogatepass error handler to return a non-empty data when decode a low surrogate with final=False. It is not necessary, but it makes all UTF-* decoders consistent and makes tests simpler.

New changeset 894263ba80af4b7733c2df95b527e96953922656 by Serhiy Storchaka in branch 'master':
bpo-24214: Fixed the UTF-8 and UTF-16 incremental decoders. (GH-14304)
https://github.com/python/cpython/commit/894263ba80af4b7733c2df95b527e96953922656

New changeset d32594ad27f48a898d42a0ea30b9d007b1c57de9 by Miss Islington (bot) in branch '3.8':
bpo-24214: Fixed the UTF-8 and UTF-16 incremental decoders. (GH-14304)
https://github.com/python/cpython/commit/d32594ad27f48a898d42a0ea30b9d007b1c57de9

New changeset c755ca89c75252a7aae9beae82fd47787a76b9e2 by Victor Stinner (Miss Islington (bot)) in branch '3.7':
[3.7] bpo-24214: Fixed the UTF-8 and UTF-16 incremental decoders. (GH-14304) (GH-14369)
https://github.com/python/cpython/commit/c755ca89c75252a7aae9beae82fd47787a76b9e2

Thanks Karthikeyan Singaravelan for the bug report of the regression. You're right that I misunderstood it. Thanks Serhiy for the second fix. The regression should now be fixed as well, I close the issue again.

Since the latest fix (GH-14369) addresses a regression between 3.7.3 and 3.7.4rc1, I am going to cherry-pick it into 3.7.4 final.  (In retrospect, this regression should have been marked as a "release blocker".)

Thanks Ned, is there a general policy on regressions to be marked as release blocker, like if a regression that was made in 3.7.3 then it acts as a release blocker for 3.7.4 or is it based on severity? ?

"The only changes allowed to occur in a maintenance branch without debate are bug fixes. Also, a general rule for maintenance branches is that compatibility must not be broken at any point between sibling minor releases (3.5.1, 3.5.2, etc.). For both rules, only rare exceptions are accepted and must be discussed first."

https://devguide.python.org/devcycle/#maintenance-branches

The principle here is that we "promise" users that they can upgrade from any version of 3.n.x to the latest version of 3.n without needing to make any changes except in very rare cases when there is an overriding concern and which must be well-documented in the release materials.  We're human so we sometimes slip up and inadvertently break that promise but that's the goal. And it's because of that promise that we can take the approach of immediately obsoleting previous older micro releases when a new micro release occurs, i.e. we don't provide fixes for 3.7.2 once 3.7.3 is released.

So, from my perspective, pretty much *any* regression between micro releases is a release blocker but especially in a case like this where it can be addressed before a final release.  That's basically why we do release candidates :)

Thanks Ned for the details, in future reports I will also try to add the respective release manager to the issue in case of regressions.

@xtreak, If you set the priority to "release blocker", the appropriate release managers will be nosied automatically.  I'd rather see too many potential "release blocker"s than too few.

New changeset 30c2ae4dcfd19acbdfb7045151c73d5700eec7b4 by Ned Deily (Miss Islington (bot)) in branch '3.7':
[3.7] bpo-24214: Fixed the UTF-8 and UTF-16 incremental decoders. (GH-14304) (GH-14369)
https://github.com/python/cpython/commit/30c2ae4dcfd19acbdfb7045151c73d5700eec7b4