classification
Title: ctypes string pointer fields should accept embedded null characters
Type: behavior Stage: resolved
Components: ctypes Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: ZackerySpytz, amaury.forgeotdarc, belopolsky, eryksun, lukasz.langa, meador.inge, miss-islington, ned.deily, serhiy.storchaka, theller
Priority: critical Keywords: 3.6regression, patch

Created on 2018-02-01 19:54 by theller, last changed 2021-05-02 11:03 by lukasz.langa. This issue is now closed.

Files
File name Uploaded Description Edit
nullchars.py theller, 2018-02-01 19:54
Pull Requests
URL Status Linked Edit
PR 8721 merged ZackerySpytz, 2018-08-10 05:50
PR 25811 merged miss-islington, 2021-05-02 10:40
PR 25812 merged miss-islington, 2021-05-02 10:40
Messages (7)
msg311462 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2018-02-01 19:54
ctypes Structure fields of type c_char_p or c_wchar_p used to accept strings with embedded null characters.  I noticed that Python 3.6.4 does refuse them.  It seems this has been changed in recent version(s).

There ARE use-cases for this:  The Windows-API OPENFILENAME structure is one example.  The Microsoft docs for the lpstrFilter field:

"""
lpstrFilter

    Type: LPCTSTR

    A buffer containing pairs of null-terminated filter strings. The last string in the buffer must be terminated by two NULL characters.
"""

I have attached a simple script which demonstrates this new behaviour; the output with Python 3.6.4 is this:

Traceback (most recent call last):
  File "nullchars.py", line 8, in <module>
    t.unicode = u"foo\0bar"
ValueError: embedded null character
msg311468 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2018-02-01 20:51
PyUnicode_AsWideCharString was updated to raise ValueError for embedded nulls if the `size` output parameter is NULL. Z_set in cfield.c should be updated to get the size, which can be ignored here. For example:

    Py_ssize_t size; 
    buffer = PyUnicode_AsWideCharString(value, &size);
msg314567 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-03-28 07:14
The change mentioned was made in GH-2462 for Issue13617 and was released in 3.6.3 (and 3.5.4 now in security-fix-only mode).
msg314579 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-03-28 09:07
This is a regression. Eryk's solution LGTM. Do you mind to create a PR?

But u"foo\0bar" is not terminated by two NULL characters. If this is used in real code, it contains a bug. And the getter of this field will return the string only to the first null character. More work is needed for making this more reliable.
msg392685 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-05-02 10:40
New changeset 73766b0341674f3920f4ea86a6f8288b801960f9 by Zackery Spytz in branch 'master':
bpo-32745: Fix a regression in the handling of ctypes' c_wchar_p type (#8721)
https://github.com/python/cpython/commit/73766b0341674f3920f4ea86a6f8288b801960f9
msg392688 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-05-02 11:03
New changeset cf6a79644c227159b8b5a44055650266d578b9f6 by Miss Islington (bot) in branch '3.9':
bpo-32745: Fix a regression in the handling of ctypes' c_wchar_p type (GH-8721) (#25812)
https://github.com/python/cpython/commit/cf6a79644c227159b8b5a44055650266d578b9f6
msg392689 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-05-02 11:03
New changeset db3ce79469ce9f8168ea4ac3e186be8e3fe44105 by Miss Islington (bot) in branch '3.8':
bpo-32745: Fix a regression in the handling of ctypes' c_wchar_p type (GH-8721) (#25811)
https://github.com/python/cpython/commit/db3ce79469ce9f8168ea4ac3e186be8e3fe44105
History
Date User Action Args
2021-05-02 11:03:54lukasz.langasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-05-02 11:03:26lukasz.langasetmessages: + msg392689
2021-05-02 11:03:13lukasz.langasetmessages: + msg392688
2021-05-02 10:40:30miss-islingtonsetpull_requests: + pull_request24499
2021-05-02 10:40:25miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request24498
2021-05-02 10:40:25lukasz.langasetnosy: + lukasz.langa
messages: + msg392685
2021-03-19 04:37:09eryksunsetversions: + Python 3.9, Python 3.10, - Python 3.6, Python 3.7
2018-08-10 05:52:17ZackerySpytzsetnosy: + ZackerySpytz
2018-08-10 05:50:54ZackerySpytzsetkeywords: + patch
stage: test needed -> patch review
pull_requests: + pull_request8206
2018-03-28 09:07:54serhiy.storchakasetmessages: + msg314579
2018-03-28 07:14:34ned.deilysetpriority: normal -> critical
nosy: + belopolsky, amaury.forgeotdarc, meador.inge, serhiy.storchaka, ned.deily
messages: + msg314567

2018-02-01 20:51:07eryksunsetversions: + Python 3.7, Python 3.8
nosy: + eryksun

messages: + msg311468

type: behavior
stage: test needed
2018-02-01 19:54:50thellercreate