This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: PyUnicode_Decode breaks when Python / sqlite3 is built with sqlite 3.12.0
Type: Stage:
Components: Extension Modules Versions: Python 2.7
process
Status: closed Resolution: works for me
Dependencies: Superseder:
Assigned To: Nosy List: zzzeek
Priority: normal Keywords:

Created on 2016-04-04 19:25 by zzzeek, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (2)
msg262864 - (view) Author: mike bayer (zzzeek) * Date: 2016-04-04 19:25
So I really don't know *where* the issue is in this one, because I don't know enough about the different bits.

Step 1:  Save this C program to demo.c:

#include <Python.h>

static PyObject *
unicode_thing(PyObject *self, PyObject *value)
{
    char *str;
    Py_ssize_t len;

    if (value == Py_None)
        Py_RETURN_NONE;

    if (PyString_AsStringAndSize(value, &str, &len))
        return NULL;

   return PyUnicode_Decode(str, len, "utf-8", "ignore");
}


static PyMethodDef UnicodeMethods[] = {
    {"unicode_thing",  unicode_thing, METH_O,
     "do a unicode thing."},
    {NULL, NULL, 0, NULL}        /* Sentinel */
};



PyMODINIT_FUNC
initdemo(void)
{
    (void) Py_InitModule("demo", UnicodeMethods);
}


Step 2:  Build with a setup.py:

from distutils.core import setup, Extension

module1 = Extension('demo',
                    sources = ['demo.c'])

setup (name = 'PackageName',
       version = '1.0',
       description = 'This is a demo package',
       ext_modules = [module1])


$ python setup.py build_ext

3. Run with a normal Python that is *not* built against SQLite 3.12.0:

$ python
Python 2.7.10 (default, Sep  8 2015, 17:20:17) 
[GCC 5.1.1 20150618 (Red Hat 5.1.1-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import demo
>>> demo.unicode_thing('sql')
u'sql'

4. Now build Python 2.7.11 *with* SQLite 3.12.0 in the -I / -L paths.  Run the same program *With* that Python:

$ /opt/Python-2.7.11-sqlite-3.12.0/bin/python
Python 2.7.11 (default, Apr  4 2016, 14:20:47) 
[GCC 5.3.1 20151207 (Red Hat 5.3.1-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import demo
>>> demo.unicode_thing('sql')
u's\x00q'

Somehow the presence of sqlite-3.12.0 in the build is breaking the Python interpreter.   I think.  I really don't know.   The bad news is, this is the code for SQLAlchemy's unicode processor, and as SQlite 3.12.0 starts getting put in distros, the world is going to break.  So this is kind of really hard to test, I don't understand it, and it's totally urgent.  Any insights would be appreciated!
msg262865 - (view) Author: mike bayer (zzzeek) * Date: 2016-04-04 19:38
i realized this is probably with my build overall.  let me do some more testing and ill reopen if i can confirm this more closely.
History
Date User Action Args
2022-04-11 14:58:29adminsetgithub: 70877
2016-04-04 19:38:05zzzeeksetstatus: open -> closed
resolution: works for me
messages: + msg262865
2016-04-04 19:25:58zzzeekcreate