Title: BytesIO methods don't accept integer types, while StringIO counterparts do
Type: behavior Stage: resolved
Components: IO Versions: Python 3.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Oren Milman, amaury.forgeotdarc, benjamin.peterson, flox, martin.panter, pitrou, serhiy.storchaka, steve.dower, stutzbach
Priority: normal Keywords: patch

Created on 2017-03-06 21:27 by Oren Milman, last changed 2017-08-26 14:52 by steve.dower. This issue is now closed.

File name Uploaded Description Edit
patchDraft1.diff Oren Milman, 2017-03-08 10:05 patch first draft review
Pull Requests
URL Status Linked Edit
PR 560 merged Oren Milman, 2017-03-08 12:35
PR 606 merged Oren Milman, 2017-03-10 22:28
Messages (19)
msg289136 - (view) Author: Oren Milman (Oren Milman) * Date: 2017-03-06 21:27
------------ current state ------------
import io

class IntLike():
    def __init__(self, num):
        self._num = num

    def __index__(self):
        return self._num

    __int__ = __index__

io.StringIO('blah blah').read(IntLike(2))
io.StringIO('blah blah').readline(IntLike(2))
io.StringIO('blah blah').truncate(IntLike(2))

io.BytesIO(b'blah blah').read(IntLike(2))
io.BytesIO(b'blah blah').readline(IntLike(2))
io.BytesIO(b'blah blah').truncate(IntLike(2))

The three StringIO methods are called without any error, but each of the three
BytesIO methods raises a "TypeError: integer argument expected, got 'IntLike'".

This is because the functions which implement the StringIO methods (in
    - _io_StringIO_read_impl
    - _io_StringIO_readline_impl
    - _io_StringIO_truncate_impl
use PyNumber_AsSsize_t, which might call nb_index.

However, the functions which implement the BytesIO methods (in
    - _io_BytesIO_read_impl
    - _io_BytesIO_readline_impl
    - _io_BytesIO_truncate_impl
use PyLong_AsSsize_t, which accepts only Python ints (or objects whose type is
a subclass of int).

------------ proposed changes ------------
- change those BytesIO methods so that they would accept integer types (i.e.
  classes that define __index__), mainly by replacing PyLong_AsSsize_t with
- add tests to Lib/test/ to verify that all six aforementioned
  methods accept integer types
msg289144 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2017-03-06 23:10
What is the use case? Unless changing the behaviour would be useful, I think the simplest solution would be to document that the methods should only be given instances of “int”, so that it is clear that other kinds of numbers are unsupported.
msg289148 - (view) Author: Oren Milman (Oren Milman) * Date: 2017-03-07 01:13
I don't have a use case for that. (I noticed this behavior by
chance, while working on some other issue.)

However, IIUC, commit 4fa88fa0ba35e25ad9be66ebbdaba9aca553dc8b,
by Benjamin Peterson, Antoine Pitrou and Amaury Forgeot d'Arc,
includes patching the aforementioned StringIO functions, so that
they would accept integer types.
So I add them to the nosy list, as I guess that the use cases for
accepting integer types in StringIO methods are also use cases
for accepting integer types in BytesIO.

And now that you mention the docs, according to them, both StringIO
and BytesIO inherit these methods from BufferedIOBase or IOBase.
Thus, the methods are already expected to behave the same, aren't
msg289151 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-07 06:18
Other objects in the io module use special-purposed converter _PyIO_ConvertSsize_t() which checks PyNumber_Check() and calls PyNumber_AsSsize_t().

I think StringIO implementation can be changed to reuse _PyIO_ConvertSsize_t() for simplicity.

After that BytesIO implementation can be changed to use the same converter just for uniformity.
msg289154 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-07 06:32
Are there tests for accepting non-integers in other classes? If no then don't bother about tests. I suppose all this came from the implementation of the 'n' format unit in PyArg_Parse* functions.
msg289158 - (view) Author: Oren Milman (Oren Milman) * Date: 2017-03-07 10:43
using _PyIO_ConvertSsize_t sounds great.

with regard to tests for accepting integer types in other classes,
there aren't a lot of them, so I guess it is not always tested.
still, it is tested in (excluding tests of the actual feature of
treating integer types as ints): test_bytes, test_cmath, test_int,
test_math, test_range, test_re, test_slice, test_struct,
test_unicode and test_weakref.
msg289160 - (view) Author: Oren Milman (Oren Milman) * Date: 2017-03-07 11:32
also with regard to adding tests, consider the following scenario:
in the future, someone writes a patch for these functions, which
makes them not accept integer types anymore.
assuming the tests don't exist, the writer of the patch doesn't
realize the patch broke something, and so the patch is committed.
years later, when the patch is finally a part of the stable release,
it breaks a lot of code out there.

lastly, ISTM adding these tests would be quite simple anyway.
msg289225 - (view) Author: Oren Milman (Oren Milman) * Date: 2017-03-08 10:05
I wrote a patch, but I attach it here and not in a PR, as you haven't approved
adding tests. (I would be happy to open a PR with or without the tests.)
I ran the test module (on my Windows 10), and seems like the patch doesn't
break anything.

also, while running test_memoryio with my added tests, i noticed some places in
Lib/ which seemed like they should be changed.
in particular, I changed 'var.__index__' to 'var = var.__index__()' in some
I feel really uncomfortable about that change, as it undos a change committed
by Florent Xicluna in b14930cd93e74cae3b7370262c6dcc7c28e0e712.
Florent, what was the reason for that change?
msg289226 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-08 10:24
msg289227 - (view) Author: Oren Milman (Oren Milman) * Date: 2017-03-08 10:31
should I open a PR (even though we didn't give Florent enough time to
msg289228 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-08 10:37
Open a PR. It will be not hard to make small changes after opening it.
msg289229 - (view) Author: Oren Milman (Oren Milman) * Date: 2017-03-08 10:40

In general, should drafts (like this one) be uploaded here?
or is it always better to open a PR?
msg289230 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-08 10:58
This for your preference.
msg289359 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-10 13:07
I would accept changes to Modules/_io/ because I consider them as code cleanup (with little side effect). But I'm not sure about changes to Python implementation and tests. Could you create more narrow PR for Modules/_io/ changes and left other changes for the consideration of the io module maintainers?
msg289414 - (view) Author: Oren Milman (Oren Milman) * Date: 2017-03-10 22:33
msg289416 - (view) Author: Oren Milman (Oren Milman) * Date: 2017-03-10 22:55
thanks :)
I would update the original PR soon.
msg289418 - (view) Author: Oren Milman (Oren Milman) * Date: 2017-03-10 23:05
or maybe git is smart enough to realize what happened, and I don't have 
to update PR 560?
msg290235 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-24 22:28
New changeset 740025478dcd0e9e4028507f32375c85f849fb07 by Serhiy Storchaka (orenmn) in branch 'master':
bpo-29741: Clean up C implementations of BytesIO and StringIO. (#606)
msg300796 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-08-24 18:33
New changeset de50360ac2fec81dbf733f6c3c739b39a8822a39 by Steve Dower (Oren Milman) in branch 'master':
bpo-29741: Update some methods in the _pyio module to also accept integer types. Patch by Oren Milman. (#560)
Date User Action Args
2017-08-26 14:52:50steve.dowersetstatus: open -> closed
resolution: fixed
stage: resolved
2017-08-24 18:33:44steve.dowersetnosy: + steve.dower
messages: + msg300796
2017-03-24 22:28:44serhiy.storchakasetmessages: + msg290235
2017-03-10 23:05:20Oren Milmansetmessages: + msg289418
2017-03-10 22:55:15Oren Milmansetmessages: + msg289416
2017-03-10 22:33:19Oren Milmansetmessages: + msg289414
2017-03-10 22:28:02Oren Milmansetpull_requests: + pull_request501
2017-03-10 13:07:24serhiy.storchakasetmessages: + msg289359
2017-03-08 12:35:47Oren Milmansetpull_requests: + pull_request461
2017-03-08 10:58:53serhiy.storchakasetmessages: + msg289230
2017-03-08 10:40:16Oren Milmansetmessages: + msg289229
2017-03-08 10:37:35serhiy.storchakasetmessages: + msg289228
2017-03-08 10:31:34Oren Milmansetmessages: + msg289227
2017-03-08 10:24:53serhiy.storchakasetnosy: + stutzbach
messages: + msg289226
2017-03-08 10:05:44Oren Milmansetfiles: + patchDraft1.diff

nosy: + flox
messages: + msg289225

keywords: + patch
2017-03-07 11:32:46Oren Milmansetmessages: + msg289160
2017-03-07 10:43:50Oren Milmansetmessages: + msg289158
2017-03-07 06:32:26serhiy.storchakasetmessages: + msg289154
2017-03-07 06:18:03serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg289151
2017-03-07 01:13:10Oren Milmansetnosy: + amaury.forgeotdarc, pitrou, benjamin.peterson
messages: + msg289148
2017-03-06 23:10:34martin.pantersetnosy: + martin.panter
messages: + msg289144
2017-03-06 21:27:25Oren Milmancreate