classification
Title: lib2to3 generates invalid code with filter and ternary operator
Type: behavior Stage: resolved
Components: 2to3 (2.x to 3.x conversion tool) Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Zoran Simic, benjamin.peterson, corona10, miss-islington, vstinner
Priority: normal Keywords: patch

Created on 2019-11-20 23:12 by Zoran Simic, last changed 2020-01-07 17:52 by miss-islington. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 17780 merged corona10, 2020-01-01 09:47
PR 17899 merged miss-islington, 2020-01-07 17:31
PR 17900 merged miss-islington, 2020-01-07 17:31
Messages (7)
msg357112 - (view) Author: Zoran Simic (Zoran Simic) Date: 2019-11-20 23:12
This code snippet exposes a small edge case in lib2to3, where syntactically invalid code is generated:

data = [1, 2, 3, 4, 5]
x = filter(lambda x: True if x > 2 else False, data)
print(x)


lib2to3 transforms 2nd line above to:
x = [x for x in data if True if x > 2 else False]


which is invalid (would be OK with parentheses after 1st 'if')

Admittedly, the original code here is more complex that it should be ('True if foo else False' being equivalent to just 'foo'), but walked into this in "the real world" and wanted to report it.
msg357133 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2019-11-21 06:13
I can reproduce on the latest master branch.

./python Tools/scripts/2to3 2.py
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Refactored 2.py
--- 2.py	(original)
+++ 2.py	(refactored)
@@ -1,3 +1,3 @@
 data = [1, 2, 3, 4, 5]
-x = filter(lambda x: True if x > 2 else False, data)
+x = [x for x in data if True if x > 2 else False]
 print(x)
RefactoringTool: Files that need to be modified:
RefactoringTool: 2.py
msg357384 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2019-11-23 19:41
Dear core developers,

I 'd like to discuss fixing this issue.
IMHO, There are 2 options to fix this issue.

1. Add parenthesize when creating ListComp.
The change will be as follow:
             new = ListComp(results.get("fp").clone(),
                            results.get("fp").clone(),
                            results.get("it").clone(),
-                           results.get("xp").clone())
+                           parenthesize(results.get("xp").clone()))

But generated codes are not pretty.

2. Transform the lambda function as same as a normal function.
The only needed is removing lambda specific logics on fixer.

The generated code will be like this in this case.
x = filter(lambda x: True if x > 2 else False, data)
x = list(filter(lambda x: True if x > 2 else False, data))

Personally, I prefer the 2nd option because it's more clear.
If the proposal is accepted, I 'd like to fix this issue. 

If there are any ideas, please let me know.
Thank you always!
msg359525 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-07 17:31
New changeset b821173b5458d137c8d5edb6e9b4997aac800a38 by Victor Stinner (Dong-hee Na) in branch 'master':
bpo-38871: Fix lib2to3 for filter-based statements that contain lambda (GH-17780)
https://github.com/python/cpython/commit/b821173b5458d137c8d5edb6e9b4997aac800a38
msg359526 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-07 17:32
Thanks Zoran Simic to the bug report, and thanks Dong-hee Na for fixing it.

It's fixed in the master branch and backports to 3.7 and 3.8 will land as soon as the CI tests pass.
msg359529 - (view) Author: miss-islington (miss-islington) Date: 2020-01-07 17:47
New changeset 535a3c4e3da2f0076bd62f04fb2cc44999fc2419 by Miss Islington (bot) in branch '3.7':
bpo-38871: Fix lib2to3 for filter-based statements that contain lambda (GH-17780)
https://github.com/python/cpython/commit/535a3c4e3da2f0076bd62f04fb2cc44999fc2419
msg359530 - (view) Author: miss-islington (miss-islington) Date: 2020-01-07 17:52
New changeset 39a5c889d30d03a88102e56f03ee0c95db198fb3 by Miss Islington (bot) in branch '3.8':
bpo-38871: Fix lib2to3 for filter-based statements that contain lambda (GH-17780)
https://github.com/python/cpython/commit/39a5c889d30d03a88102e56f03ee0c95db198fb3
History
Date User Action Args
2020-01-07 17:52:11miss-islingtonsetmessages: + msg359530
2020-01-07 17:47:57miss-islingtonsetnosy: + miss-islington
messages: + msg359529
2020-01-07 17:32:59vstinnersetstatus: open -> closed
versions: + Python 3.9
messages: + msg359526

resolution: fixed
stage: patch review -> resolved
2020-01-07 17:31:14miss-islingtonsetpull_requests: + pull_request17310
2020-01-07 17:31:08miss-islingtonsetpull_requests: + pull_request17309
2020-01-07 17:31:01vstinnersetnosy: + vstinner
messages: + msg359525
2020-01-01 09:47:08corona10setkeywords: + patch
stage: patch review
pull_requests: + pull_request17213
2019-11-23 19:41:58corona10setnosy: + benjamin.peterson
messages: + msg357384
2019-11-21 06:13:18corona10setnosy: + corona10
messages: + msg357133
2019-11-20 23:12:36Zoran Simiccreate