Author eryksun
Recipients eryksun, fooofei
Date 2017-06-12.10:58:20
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1497265102.35.0.73914586969.issue30634@psf.upfronthosting.co.za>
In-reply-to
Content
It's undocumented that cast() should work to directly convert Python strings to pointers. Even when it seems to work, it's a risky thing to depend on because there's no source ctypes data object to reference. Thus there's neither _b_base_ nor anything in _objects to support the reference. If the string has since been deallocated, the pointer is invalid.

What you've uncovered is an implementation detail. Windows has a 16-bit unsigned wchar_t type, so HAVE_USABLE_WCHAR_T is defined when building the default narrow build in Python 2. In this case ctypes can use PyUnicode_AS_UNICODE, which is why you can get the base address of the unicode object's internal buffer on Windows. 

Linux systems define wchar_t as a 4-byte signed value. IIRC it's a typedef for int. Because wchar_t is signed in this case, HAVE_USABLE_WCHAR_T is not defined even for a wide build. ctypes has to temporarily copy the string via PyUnicode_AsWideChar. It references the memory in a capsule object. You can see this by constructing a c_wchar_p instance, for example:

    >>> p = ctypes.c_wchar_p(u'helloworld')
    >>> p._objects
    <capsule object "_ctypes/cfield.c wchar_t buffer from unicode" at 0x7fedb67d5f90>

In your case, by the time you actually look at the address, the capsule has been deallocated, and the memory is no longer valid. For example:

    >>> addr = ctypes.cast(u'helloworld', ctypes.c_void_p).value
    >>> ctypes.wstring_at(addr, 10)
    u'\U0150ccf0\x00\U0150cc00\x00oworld'

It works as expected if one instead casts a c_wchar_p instance, which references the capsule to keep the memory alive:

    >>> addr = ctypes.cast(p, ctypes.c_void_p).value
    >>> ctypes.wstring_at(addr, 10)
    u'helloworld'

However, that's not what you want since we know it's a copy. I think your only option is to use the C API via ctypes.pythonapi. For example:

    ctypes.pythonapi.PyUnicodeUCS4_AsUnicode.argtypes = (ctypes.py_object,)
    ctypes.pythonapi.PyUnicodeUCS4_AsUnicode.restype = ctypes.c_void_p

    s = u'helloworld'
    addr = ctypes.pythonapi.PyUnicodeUCS4_AsUnicode(s)

    >>> ctypes.wstring_at(addr, 10)
    u'helloworld'

On narrow builds this function is exported a PyUnicodeUCS2_AsUnicode.
History
Date User Action Args
2017-06-12 10:58:22eryksunsetrecipients: + eryksun, fooofei
2017-06-12 10:58:22eryksunsetmessageid: <1497265102.35.0.73914586969.issue30634@psf.upfronthosting.co.za>
2017-06-12 10:58:22eryksunlinkissue30634 messages
2017-06-12 10:58:20eryksuncreate