Calling int() or long() on a buffer() object in Python 2.7 does not do the right thing. The following code snippet:

buf = buffer("123test", 1, 2)
print buf
print int(buf)

does not do what I would expect (that it print "23" twice). Instead, it prints "23" once and then throws an exception:

ValueError: invalid literal for int() with base 10: '23test'

This is caused by Objects/abstract.c function int_from_string(), which gets passed the length of the string but does not actually use that information to limit what part is parsed from the string. It only uses it to check for embedded NUL bytes. The real culprit is probably PyInt_FromString() which does not take a length indicator.

This is similar to bpo-24802. The patch for bpo-24802 was not backported to 2.7 because affected functions don't accept memoryview in Python 2. Now we have an example, and can backport that patch.

Now we have an example, and can backport that patch.

More seriously it's possible to get a buffer over-read using NumPy:

    >>> import numpy
    >>> int(buffer(numpy.array('123', dtype='c')))
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
     ValueError: invalid literal for int() with base 10: '123\xe1\x18\x7f'

I backported the modification to PyNumber_Int and PyNumber_Long, using PyString_FromStringAndSize and PyString_AS_STRING. It works as expected:

    Python 2.7.10+ (2.7:5d88c1d413b9+, Nov 20 2015, 04:58:55) 
    [GCC 4.8.4] on linux2
    Type "help", "copyright", "credits" or "license" for more 
    information.
    >>> int(buffer('123test', 1, 2))
    23
    [41951 refs]
    >>> long(buffer('123test', 1, 2))
    23L
    [41952 refs]

Do you forgot to add a patch?

I just made a quick modification to check that it works. I'm sure you could do the same. But here it is anyway.

Could you backport full patch, with tests? And compile() is affected too:

>>> compile(buffer("123\0test", 1, 2), '', 'exec')
<code object <module> at 0xb70c5800, file "", line 1>
>>> compile(buffer("123test", 1, 2), '', 'exec')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: compile() expected string without null bytes

Eryk, could the tests in the path file that you posted regarding int() and float() be incorrect in the sense that buffer(...)[a:b] returns a str() and not another buffer() (and will thus always be NUL terminated and not exhibit the problem, whether or not the C source have been patch)? I think the tests should be using buffer(..., a, b) instead.

May I say that I am very, very impressed by the speed with which this has been picked up? Thank you all for your impressive work!

(Please excuse my horrible spelling in my last message, I'm apparently more tired than I care to admit)

I think the tests should be using buffer(..., a, b) instead.

Thanks, you're right. :)

New changeset 3ef7d1af5195 by Serhiy Storchaka in branch '2.7':
Issue bpo-25678: Copy buffer objects to null-terminated strings.
https://hg.python.org/cpython/rev/3ef7d1af5195

Committed after fixing some details. Thank you Eryk Sun.