classification
Title: unknown error handlers should be reported early
Type: Stage: resolved
Components: Unicode Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: ezio.melotti, hroncok, methane, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2019-06-24 14:10 by hroncok, last changed 2020-04-07 14:07 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
bench.py vstinner, 2019-06-24 15:10
Pull Requests
URL Status Linked Edit
PR 14341 merged vstinner, 2019-06-24 14:33
PR 14385 merged vstinner, 2019-06-25 23:28
PR 19409 merged vstinner, 2020-04-07 13:45
Messages (8)
msg346407 - (view) Author: Miro Hrončok (hroncok) * Date: 2019-06-24 14:10
I was just bit by specifying an nonexisitng error handler for bytes.decode() without noticing.

Consider this code:

>>> 'a'.encode('cp1250').decode('utf-8', errors='Boom, Shaka Laka, Boom!')
'a'

Nobody notices that the error handler doesn't exist.

However:

>>> 'ž'.encode('cp1250').decode('utf-8', errors='Boom, Shaka Laka, Boom!')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: unknown error handler name 'Boom, Shaka Laka, Boom!'


The error is only noticeable once there is an error in the data.

While nobody could possibly mistake 'Boom, Shaka Laka, Boom!' for a valid error handler, I was bit by this:

>>> b.decode('utf-8', errors='surrogate')

Which in fact should have been

>>> b.decode('utf-8', errors='surrogateescape')

Yet I wasn't notified, because the bytes in question were actually decodeable as valid utf-8.

I suggest that unknown error handler should rise an exception immediately like this:

>>> 'b'.encode('cp1250').decode('utf-8', errors='Boom, Shaka Laka, Boom!')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: unknown error handler name 'Boom, Shaka Laka, Boom!'
msg346409 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-24 14:15
Getting an error handler is expensive compared to the time to encode text/decode bytes. Python loads the error handler lazily to provide best performances. Text codecs are performance critical for Python.

If we add a check, it should only be enabled in development (python3 -X dev) and/or debug mode (./configure --with-pydebug).
msg346414 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-24 15:10
bench.py: microbenchmark to measure the overhead of PR 14341.
msg346569 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-25 22:51
New changeset 22eb689cf3de7972a2789db3ad01a86949508ab7 by Victor Stinner in branch 'master':
bpo-37388: Development mode check encoding and errors (GH-14341)
https://github.com/python/cpython/commit/22eb689cf3de7972a2789db3ad01a86949508ab7
msg346573 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-25 23:49
New changeset ed076ed467264b43ed01a8223ca65b133b590919 by Victor Stinner in branch 'master':
bpo-37388: Add PyUnicode_Decode(str, 0) fast-path (GH-14385)
https://github.com/python/cpython/commit/ed076ed467264b43ed01a8223ca65b133b590919
msg346574 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-26 00:12
I compared ref (commit e1a63c4f21011a3ae77dff624196561070c83446) to patch (commit ed076ed467264b43ed01a8223ca65b133b590919). I ran bench.py using:

# edit Makefile.pre.in to use:
PROFILE_TASK=-m test.regrtest --pgo test_unicode test_codecs

./configure --enable-optimizations && make
./python -m venv env && env/bin/python -m pip install -U pyperf
./env/bin/python bench.py -o <file>.json

$ python3 -m pyperf compare_to ref_e1a63c4f2.json patch_ed076ed4.json 
0B: Mean +- std dev: [ref_e1a63c4f2] 76.7 ns +- 0.9 ns -> [patch_ed076ed4] 77.5 ns +- 2.9 ns: 1.01x slower (+1%)
1B: Mean +- std dev: [ref_e1a63c4f2] 92.9 ns +- 0.8 ns -> [patch_ed076ed4] 94.0 ns +- 2.4 ns: 1.01x slower (+1%)
5B: Mean +- std dev: [ref_e1a63c4f2] 106 ns +- 2 ns -> [patch_ed076ed4] 110 ns +- 2 ns: 1.04x slower (+4%)
10B: Mean +- std dev: [ref_e1a63c4f2] 105 ns +- 1 ns -> [patch_ed076ed4] 109 ns +- 1 ns: 1.03x slower (+3%)
25B: Mean +- std dev: [ref_e1a63c4f2] 108 ns +- 3 ns -> [patch_ed076ed4] 111 ns +- 3 ns: 1.03x slower (+3%)
100B: Mean +- std dev: [ref_e1a63c4f2] 114 ns +- 1 ns -> [patch_ed076ed4] 115 ns +- 2 ns: 1.01x slower (+1%)
1000B: Mean +- std dev: [ref_e1a63c4f2] 267 ns +- 3 ns -> [patch_ed076ed4] 253 ns +- 4 ns: 1.06x faster (-5%)

$ python3 -m pyperf compare_to ref_e1a63c4f2.json patch_ed076ed4.json  --table
+-----------+---------------+-----------------------------+
| Benchmark | ref_e1a63c4f2 | patch_ed076ed4              |
+===========+===============+=============================+
| 0B        | 76.7 ns       | 77.5 ns: 1.01x slower (+1%) |
+-----------+---------------+-----------------------------+
| 1B        | 92.9 ns       | 94.0 ns: 1.01x slower (+1%) |
+-----------+---------------+-----------------------------+
| 5B        | 106 ns        | 110 ns: 1.04x slower (+4%)  |
+-----------+---------------+-----------------------------+
| 10B       | 105 ns        | 109 ns: 1.03x slower (+3%)  |
+-----------+---------------+-----------------------------+
| 25B       | 108 ns        | 111 ns: 1.03x slower (+3%)  |
+-----------+---------------+-----------------------------+
| 100B      | 114 ns        | 115 ns: 1.01x slower (+1%)  |
+-----------+---------------+-----------------------------+
| 1000B     | 267 ns        | 253 ns: 1.06x faster (-5%)  |
+-----------+---------------+-----------------------------+

The overhead of my change is around 1 ns, 4 ns (on 106 ns) in the worst case (5B).

The change "looks" faster on the 1000B case, but it's likely a glitch of PGO compilation which is not really deterministic.
msg346575 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-26 00:13
Ok, the encoding and errors are now checked in almost all cases if you enable the development mode using -X dev command line option.
msg365906 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-07 14:07
New changeset d8acf0d9aae71d1897e8f91989bd8bfb4a9ef9c6 by Victor Stinner in branch 'master':
bpo-37388: Don't check encoding/errors during finalization (GH-19409)
https://github.com/python/cpython/commit/d8acf0d9aae71d1897e8f91989bd8bfb4a9ef9c6
History
Date User Action Args
2020-04-07 14:07:50vstinnersetmessages: + msg365906
2020-04-07 13:45:24vstinnersetpull_requests: + pull_request18770
2019-06-26 00:13:23vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg346575

stage: patch review -> resolved
2019-06-26 00:12:39vstinnersetmessages: + msg346574
2019-06-25 23:49:35vstinnersetmessages: + msg346573
2019-06-25 23:28:20vstinnersetpull_requests: + pull_request14199
2019-06-25 22:51:08vstinnersetmessages: + msg346569
2019-06-24 15:10:19vstinnersetfiles: + bench.py

messages: + msg346414
2019-06-24 14:33:51vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request14162
2019-06-24 14:15:19vstinnersetnosy: + methane, serhiy.storchaka
2019-06-24 14:15:11vstinnersetmessages: + msg346409
2019-06-24 14:10:52hroncokcreate