classification
Title: PyString_FromFormatV() fails to build empty strings
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Tey, miss-islington, serhiy.storchaka
Priority: normal Keywords: patch, patch, patch

Created on 2018-06-09 20:36 by Tey, last changed 2019-01-12 07:48 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 11515 merged serhiy.storchaka, 2019-01-11 09:19
PR 11515 merged serhiy.storchaka, 2019-01-11 09:19
PR 11515 merged serhiy.storchaka, 2019-01-11 09:19
PR 11516 merged serhiy.storchaka, 2019-01-11 09:20
PR 11516 merged serhiy.storchaka, 2019-01-11 09:20
PR 11516 merged serhiy.storchaka, 2019-01-11 09:20
PR 11532 merged miss-islington, 2019-01-12 07:22
PR 11532 merged miss-islington, 2019-01-12 07:22
Messages (6)
msg319180 - (view) Author: Tey (Tey) Date: 2018-06-09 20:36
Making PyString_FromFormatV() build an empty string on 2.7 will fail and raise the following error:
SystemError: Objects/stringobject.c:3903: bad argument to internal function

It does not matter if format is empty or not (e.g., both PyString_FromFormat("") and PyString_FromFormat("%s", "")).

The exception is raised from _PyString_Resize() (called at the end of PyString_FromFormatV()), because the string given to that method has a ref count different than 1. This is expected for empty strings, because PyString_FromStringAndSize() returns a reference to <nullstring> when <size> is 0.

A possible fix would be to prevent the call to _PyString_Resize() from PyString_FromFormatV() if <string> is equal to <nullstring>, or if there is no need to resize the string.

Python 3 versions before 3.6 might be impacted as well through PyBytes_FromFormatV() (cannot check).

The following code can be used to trigger the bug:

    static int dobug(const char *fmt, ...) {
        va_list args;
        va_start(args, fmt);
    
        PyObject *str = PyString_FromFormatV(fmt, args);
        va_end(args);
    
        if(str == NULL) {
            fprintf(stderr, "Error: PyString_FromFormatV(%s) returned NULL\n", fmt);
            return -1;
        }
    
        Py_DECREF(str);
    
        return 0;
    }

    static PyObject* bug(PyObject *self) {
        fprintf(stderr, "dobug(\"\") => %d\n", dobug(""));
        fprintf(stderr, "dobug(\"%%s\", \"\") => %d\n", dobug("%s", ""));
    
        if(PyErr_Occurred()) return NULL;
        Py_RETURN_NONE;
    }
msg319226 - (view) Author: Tey (Tey) Date: 2018-06-10 17:43
For the record, it does not fail on 3.x because _PyBytes_Resize() checks if the "string" needs to be resized (and returns if not) before checking its ref count. Maybe something similar should be done in _PyString_Resize() for 2.x.
msg319227 - (view) Author: Tey (Tey) Date: 2018-06-10 17:53
BTW, problem does appear in 3.4 as it's only been fixed in 3.5+ as a fix for issue #25270
msg333518 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-01-12 07:22
New changeset 44cc4822bb3799858201e61294c5863f93ec12e2 by Serhiy Storchaka in branch 'master':
bpo-33817: Fix _PyBytes_Resize() for empty bytes object. (GH-11516)
https://github.com/python/cpython/commit/44cc4822bb3799858201e61294c5863f93ec12e2
msg333519 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-01-12 07:22
New changeset 08a81df05004147ee174ece645679576ab867860 by Serhiy Storchaka in branch '2.7':
bpo-33817: Fix _PyString_Resize() and _PyUnicode_Resize() for empty strings. (GH-11515)
https://github.com/python/cpython/commit/08a81df05004147ee174ece645679576ab867860
msg333520 - (view) Author: miss-islington (miss-islington) Date: 2019-01-12 07:40
New changeset d39c19255910b9dce08c595f511597e98b09e91f by Miss Islington (bot) in branch '3.7':
bpo-33817: Fix _PyBytes_Resize() for empty bytes object. (GH-11516)
https://github.com/python/cpython/commit/d39c19255910b9dce08c595f511597e98b09e91f
History
Date User Action Args
2019-01-12 07:48:00serhiy.storchakasetkeywords: patch, patch, patch
status: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-01-12 07:40:15miss-islingtonsetnosy: + miss-islington
messages: + msg333520
2019-01-12 07:22:55serhiy.storchakasetmessages: + msg333519
2019-01-12 07:22:51miss-islingtonsetpull_requests: + pull_request11130
2019-01-12 07:22:47miss-islingtonsetpull_requests: + pull_request11129
2019-01-12 07:22:36serhiy.storchakasetmessages: + msg333518
2019-01-11 09:20:35serhiy.storchakasetpull_requests: + pull_request11084
2019-01-11 09:20:30serhiy.storchakasetpull_requests: + pull_request11083
2019-01-11 09:20:26serhiy.storchakasetpull_requests: + pull_request11082
2019-01-11 09:20:05serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request11081
2019-01-11 09:20:00serhiy.storchakasetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11080
2019-01-11 09:19:55serhiy.storchakasetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11079
2018-06-10 17:53:53Teysetmessages: + msg319227
2018-06-10 17:43:47Teysetmessages: + msg319226
2018-06-10 06:58:13serhiy.storchakasetassignee: serhiy.storchaka

nosy: + serhiy.storchaka
versions: - Python 3.4, Python 3.5
2018-06-09 20:36:16Teycreate