classification
Title: 2to3 does not wrap zip correctly
Type: behavior Stage: resolved
Components: 2to3 (2.x to 3.x conversion tool) Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Mariatta, benjamin.peterson, berker.peksag, cvk, stuarteberg
Priority: normal Keywords: patch

Created on 2016-11-30 11:04 by cvk, last changed 2017-06-16 02:57 by Mariatta. This issue is now closed.

Files
File name Uploaded Description Edit
fix-28837.patch stuarteberg, 2017-01-19 18:02 In lib2to3 fixers for map, zip, and filter, handle extra "trailers", e.g. zip()[0] review
Pull Requests
URL Status Linked Edit
PR 24 merged stuarteberg, 2017-02-15 18:30
PR 2235 merged Mariatta, 2017-06-16 02:24
Messages (8)
msg282075 - (view) Author: Carsten (cvk) Date: 2016-11-30 11:04
The following Python 2.7 code is not converted correctly to Python 3 by 2to3:

c = [(1, 10), (2, 20)]

# get a tuple with the first entries of every tuple in c,
# i.e. (1, 2)
x = zip(*c)[0]

print x


The result is


c = [(1, 10), (2, 20)]

# get a tuple with the first entries of every tuple in c,
# i.e. (1, 2)
x = zip(*c)[0]

print(x)



However running it with python3 gives the following exception

Traceback (most recent call last):
  File "result.py", line 7, in <module>
    x = zip(*c)[0]
TypeError: 'zip' object is not subscriptable



Tested with 2to3-2.7 and 2to3-3.4 on debian stable (jessie)
msg285796 - (view) Author: Stuart Berg (stuarteberg) * Date: 2017-01-19 15:08
In addition to zip(), this problem also affects map() and filter().

The problem is that the match patterns in FixZip, FixMap, and FixFilter do not allow for more than one "trailer" node.  (And even if they did, their transform() methods aren't expecting it.)

For example, in the following expression, 'zip' is followed by two 'trailers', which are '(a,b)', and [0]:

zip(a,b)[0]

... but FixZip.PATTERN only expects a single trailer (the argument list), so the presence of a second trailer prevents the match: https://git.io/vMDP9


(Here's the relevant line of the grammar: https://git.io/vMDPJ)

I've written a patch that fixes this problem for zip, map, and filter, with tests.  See attached.

BTW, this problem was previously reported in 21628, but that issue was incorrectly closed as a duplicate, so I'm submitting my patch here.
msg285823 - (view) Author: Stuart Berg (stuarteberg) * Date: 2017-01-19 17:53
Sorry for re-uploading the patch; I made some pep8 fixes.
msg287596 - (view) Author: Stuart Berg (stuarteberg) * Date: 2017-02-11 08:42
Patch submitted as github PR #24:
https://github.com/python/cpython/pull/24
msg291211 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2017-04-06 05:19
New changeset 93b4b47e3a720171d67f3b608de406aef462835c by Benjamin Peterson (Stuart Berg) in branch 'master':
bpo-28837: Fix lib2to3 handling of map/zip/filter calls when followed with a 'trailer', e.g. zip()[x] (#24)
https://github.com/python/cpython/commit/93b4b47e3a720171d67f3b608de406aef462835c
msg291863 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2017-04-19 05:48
I think this at least needs to be backported to 3.6 (or can we close the issue now?)
msg296148 - (view) Author: Mariatta Wijaya (Mariatta) * (Python committer) Date: 2017-06-16 02:56
New changeset 292b421d48ab91b0c68ea4040fc7371e7d6d610e by Mariatta in branch '3.6':
bpo-28837: Fix lib2to3 handling of map/zip/filter calls when followed with a 'trailer', e.g. zip()[x] (GH-24) (GH-2235)
https://github.com/python/cpython/commit/292b421d48ab91b0c68ea4040fc7371e7d6d610e
msg296149 - (view) Author: Mariatta Wijaya (Mariatta) * (Python committer) Date: 2017-06-16 02:57
Backported to 3.6.
History
Date User Action Args
2017-06-16 02:57:29Mariattasetstatus: open -> closed
versions: - Python 2.7
messages: + msg296149

resolution: fixed
stage: backport needed -> resolved
2017-06-16 02:56:55Mariattasetnosy: + Mariatta
messages: + msg296148
2017-06-16 02:24:54Mariattasetpull_requests: + pull_request2280
2017-04-19 05:48:04berker.peksagsetnosy: + berker.peksag

messages: + msg291863
stage: backport needed
2017-04-06 05:19:42benjamin.petersonsetmessages: + msg291211
2017-02-15 18:30:06stuartebergsetpull_requests: + pull_request85
2017-02-11 08:42:30stuartebergsetmessages: + msg287596
2017-01-19 18:02:26stuartebergsetfiles: + fix-28837.patch
2017-01-19 18:01:51stuartebergsetfiles: - fix-28837.patch
2017-01-19 17:53:48stuartebergsetfiles: - fix-28837.patch
2017-01-19 17:53:39stuartebergsetfiles: + fix-28837.patch

messages: + msg285823
2017-01-19 17:39:29stuartebergsetfiles: + fix-28837.patch
2017-01-19 17:37:46stuartebergsetfiles: - fix-28837.patch
2017-01-19 15:08:08stuartebergsetfiles: + fix-28837.patch

nosy: + stuarteberg
messages: + msg285796

keywords: + patch
2016-12-01 03:02:02xiang.zhangsetnosy: + benjamin.peterson

versions: + Python 3.6, Python 3.7, - Python 3.4
2016-11-30 11:04:58cvkcreate