This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: tuple function gives wrong answer when called on list subclass with custom __iter__
Type: behavior Stage: commit review
Components: Library (Lib) Versions: Python 3.4, Python 3.5, Python 2.7
Status: closed Resolution: fixed
Dependencies: Superseder: Concrete object C API considered harmful to subclasses of builtin types
View: 10977
Assigned To: rhettinger Nosy List: David MacIver, python-dev, r.david.murray, rhettinger, serhiy.storchaka, stub
Priority: high Keywords: easy, patch

Created on 2015-03-24 08:42 by David MacIver, last changed 2022-04-11 14:58 by admin. This issue is now closed.

File name Uploaded Description Edit David MacIver, 2015-03-24 08:42 Reproduction test case
fix_list_to_tuple.diff rhettinger, 2015-05-14 00:24 Patch without the OP's test review
fix_list_to_tuple_2.diff serhiy.storchaka, 2015-05-17 09:50 + tests review
Messages (11)
msg239098 - (view) Author: David MacIver (David MacIver) * Date: 2015-03-24 08:42
Converting a list to a tuple appears to have an optimisation that is wrong in the presence of subclassing to override __iter__. It ignores the user defined iter and uses the normal list one. I've attached a file with a test case to demonstrate this.

I've verified this on both python 2.7 and python 3.4. It's presumably also the case on everything in between.

This was found because it caused a bug with a type that pytz uses, which lazily populates the list on iteration:
msg239138 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-03-24 15:06
This is a specific instance of the general problem covered by issue 10977.
msg239152 - (view) Author: David MacIver (David MacIver) * Date: 2015-03-24 16:26
Ah, I hadn't seen that. Thanks for the link.

But... is it really? They have basically the same root cause, but the general problem seems to be hard to fix, while the specific problem here seems to be basically "don't use the concrete API here because it breaks things".
msg239154 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-03-24 16:33
Right, but unless I miss my guess (I haven't looked at the code) it isn't that the concrete api is being used by the constructor, it's that the concrete API is being used by the iterator protocol called by the constructor.  If I'm wrong it would be sensible to fix the constructor and we can reopen this.  Even if I'm right perhaps there's a way to fix the constructor, but in that case it would be best addressed in the context of that issue.
msg239161 - (view) Author: David MacIver (David MacIver) * Date: 2015-03-24 17:39
So as a data point, this problem seems to be unique to tuple. set(x), list(x), tuple(iter(x)) all seem to work as expected and respect the overridden __iter__ (set and list were both included in the test case I attached to demonstrated this, iter I just checked right now). This suggests that the problem is in what tuple is doing, not in some general iterator protocol,
msg243073 - (view) Author: Stuart Bishop (stub) Date: 2015-05-13 11:34
Can we get this reopened? As David MacIver points out, this seems entirely a wart in tuple's constructor (compared to all the other builtin types), whereas 10977 is worrying about how 3rd party code using the C API can corrupt subclasses of builtin types (a much larger scope, and much less likely to be resolved in a good way).

Does it make sense to require python code wishing to case a tuple or tuple subclass do so using tuple(list(o)), or should tuple(o) work as expected? The primary use is of course converting a mutable sequence to an immutable representation to use as a dict key.
msg243158 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2015-05-14 06:06
Added a patch.  Needs to have the OP's test case added.
msg243159 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-05-14 06:10
And it would be nice to add the same test for list, set, etc (if they don't exist).
msg243392 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-05-17 09:50
Raymond's patch LGTM. Here is updated patch with tests.
msg243428 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-05-17 21:37
New changeset b6121a4afad7 by Raymond Hettinger in branch '2.7':
Issue #23757:  Only call the concrete list API for exact lists.
msg243429 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-05-17 21:47
New changeset c79530e08985 by Raymond Hettinger in branch '3.4':
Issue #23757:  Only call the concrete list API for exact lists.
Date User Action Args
2022-04-11 14:58:14adminsetgithub: 67945
2015-05-17 21:47:59rhettingersetstatus: open -> closed
resolution: fixed
2015-05-17 21:47:12python-devsetmessages: + msg243429
2015-05-17 21:37:48python-devsetnosy: + python-dev
messages: + msg243428
2015-05-17 09:50:51serhiy.storchakasetfiles: + fix_list_to_tuple_2.diff
messages: + msg243392

assignee: rhettinger
keywords: + patch
stage: needs patch -> commit review
2015-05-14 06:10:22serhiy.storchakasetmessages: + msg243159
2015-05-14 06:06:42rhettingersetnosy: + rhettinger
messages: + msg243158
2015-05-14 05:37:55serhiy.storchakasetnosy: + serhiy.storchaka
2015-05-14 00:25:02rhettingersetkeywords: + easy, - patch
priority: normal -> high
versions: + Python 3.5
2015-05-14 00:24:29rhettingersetfiles: + fix_list_to_tuple.diff
keywords: + patch
2015-05-13 12:47:51r.david.murraysetstatus: closed -> open
type: behavior
resolution: duplicate -> (no value)
stage: resolved -> needs patch
2015-05-13 11:34:00stubsetnosy: + stub
messages: + msg243073
2015-03-24 17:39:05David MacIversetmessages: + msg239161
2015-03-24 16:33:42r.david.murraysetmessages: + msg239154
2015-03-24 16:26:35David MacIversetmessages: + msg239152
2015-03-24 15:06:24r.david.murraysetstatus: open -> closed

superseder: Concrete object C API considered harmful to subclasses of builtin types

nosy: + r.david.murray
messages: + msg239138
resolution: duplicate
stage: resolved
2015-03-24 08:42:29David MacIvercreate