classification
Title: [sqlite3] Do not truncate results of user functions and aggregates on the first NUL
Type: behavior Stage: resolved
Components: Extension Modules Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: erlendaasland Nosy List: erlendaasland, miss-islington, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2021-08-04 08:29 by erlendaasland, last changed 2021-08-06 21:08 by erlendaasland. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 27588 merged erlendaasland, 2021-08-04 08:33
PR 27611 merged miss-islington, 2021-08-05 07:22
PR 27639 merged erlendaasland, 2021-08-06 18:52
Messages (8)
msg398865 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-08-04 08:29
The third argument to sqlite3_result_text() is the length of the string passed as the second argument. Currently, we pass -1, so SQLite has to invoke strlen() to compute the length of the passed string. Suggesting to use PyUnicode_AsUTF8AndSize() iso. PyUnicode_AsUTF8() and pass the string size to avoid the superfluous strlen().

See also:
- https://sqlite.org/c3ref/result_blob.html
msg398869 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-08-04 09:01
The difference between specifying negative and non-negative third argument of sqlite3_result_text() is that in the latter case the result can contain embedded NUL characters.

Could you please add a test for string containing embedded NUL?

Letting SQLite compute string length would work incorrect in case of embedded NULs, so I think that we should raise explicit error if sz > INT_MAX.
msg398873 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-08-04 09:06
> Could you please add a test for string containing embedded NUL?

Of course, thanks for the heads up.

> I think that we should raise explicit error if sz > INT_MAX.

Yes, I thought about this. I do agree that raising OverflowError would be better. Thanks.
msg398881 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-08-04 10:03
Serhiy, I've updated the PR, if you want to take a look. Thanks for your feedback.
msg398971 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-08-05 07:22
New changeset 8f010dc920e1f6dc6a357e7cc1460a7a567c05c6 by Erlend Egeberg Aasland in branch 'main':
bpo-44822: Don't truncate `str`s with embedded NULL chars returned by `sqlite3` UDF callbacks (GH-27588)
https://github.com/python/cpython/commit/8f010dc920e1f6dc6a357e7cc1460a7a567c05c6
msg399115 - (view) Author: miss-islington (miss-islington) Date: 2021-08-06 18:00
New changeset 2b1e713f877102bbca299f0f5d7db969d78db49f by Miss Islington (bot) in branch '3.10':
bpo-44822: Don't truncate `str`s with embedded NULL chars returned by `sqlite3` UDF callbacks (GH-27588)
https://github.com/python/cpython/commit/2b1e713f877102bbca299f0f5d7db969d78db49f
msg399146 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-08-06 21:02
New changeset c352412123140e79dcce6188d17e3e6dbc3f4144 by Erlend Egeberg Aasland in branch '3.9':
[3.9] bpo-44822: Don't truncate `str`s with embedded NULL chars returned by `sqlite3` UDF callbacks (GH-27588). (GH-27639)
https://github.com/python/cpython/commit/c352412123140e79dcce6188d17e3e6dbc3f4144
msg399149 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-08-06 21:08
Thanks, Serhiy!
History
Date User Action Args
2021-08-06 21:08:39erlendaaslandsetstatus: open -> closed
resolution: fixed
messages: + msg399149

stage: patch review -> resolved
2021-08-06 21:02:24serhiy.storchakasetmessages: + msg399146
2021-08-06 18:52:57erlendaaslandsetpull_requests: + pull_request26133
2021-08-06 18:00:04miss-islingtonsetmessages: + msg399115
2021-08-05 07:22:30serhiy.storchakasetmessages: + msg398971
2021-08-05 07:22:17miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request26105
2021-08-04 10:46:38serhiy.storchakasettitle: [sqlite3] Micro-optimisation: pass string size to sqlite3_result_text() -> [sqlite3] Do not truncate results of user functions and aggregates on the first NUL
type: enhancement -> behavior
versions: + Python 3.9, Python 3.10
2021-08-04 10:03:10erlendaaslandsetmessages: + msg398881
2021-08-04 09:06:42erlendaaslandsetmessages: + msg398873
2021-08-04 09:01:35serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg398869
2021-08-04 08:33:55erlendaaslandsetkeywords: + patch
stage: patch review
pull_requests: + pull_request26090
2021-08-04 08:29:38erlendaaslandcreate