classification
Title: ast.parse gives wrong position (col_offset) for some BinOp-s
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Aivar.Annamaa, Carl.Friedrich.Bolz, benjamin.peterson, levkivskyi, miss-islington, pablogsal, sam.kimbrel, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2013-07-06 09:35 by Aivar.Annamaa, last changed 2019-07-09 19:37 by levkivskyi. This issue is now closed.

Files
File name Uploaded Description Edit
binop_bug.py Aivar.Annamaa, 2013-07-06 09:35
18374-binop-col-offset-with-test.diff sam.kimbrel, 2014-04-17 15:19 review
binop-offset.patch Carl.Friedrich.Bolz, 2019-07-05 19:32
Pull Requests
URL Status Linked Edit
PR 14607 closed Carl.Friedrich.Bolz, 2019-07-05 20:46
PR 14653 merged miss-islington, 2019-07-08 21:18
PR 14659 merged Carl.Friedrich.Bolz, 2019-07-09 09:15
PR 14672 merged miss-islington, 2019-07-09 12:20
Messages (14)
msg192394 - (view) Author: Aivar Annamaa (Aivar.Annamaa) * Date: 2013-07-06 09:35
When parsing following program:

1+2-3

ast.parse gives col_offset=3 to outermost BinOp (as shown by the attached test script). Correct col_offset would be 0 for both BinOp-s
msg216696 - (view) Author: Sam Kimbrel (sam.kimbrel) * Date: 2014-04-17 15:19
Here's a patch that corrects col_offset for binops in both the ast module and in the compiler proper. I've incorporated Aivar's test into test_ast.py; if there are test suites for compile.c please let me know and I can add something there too.
msg347294 - (view) Author: Carl Friedrich Bolz-Tereick (Carl.Friedrich.Bolz) * Date: 2019-07-04 20:15
FWIW, in my opinion the col_offsets of the two nodes should be 1 and 3, respectively (the positions of the operators).
msg347313 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-07-05 08:46
I am not sure that changes in compile.c are needed now.

Python/Python-ast.c is generated file. And perhaps it would be better to fix CST instead of AST.
msg347319 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-05 09:15
Pablo, Ivan: this old bug is able CST and AST, things that you know. You may want to have a look ;-)

Note: This bug is discussed in https://twitter.com/cfbolz/status/1146866203975139328
msg347320 - (view) Author: Aivar Annamaa (Aivar.Annamaa) * Date: 2019-07-05 09:38
> in my opinion the col_offsets of the two nodes should be 1 and 3, respectively (the positions of the operators)

This would not match the documentation:

> one can get the source segment of a one-line expression node using source_line[node.col_offset : node.end_col_offset]
msg347324 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-07-05 09:51
> in my opinion the col_offsets of the two nodes should be 1 and 3, respectively (the positions of the operators)

Such change was committed once, but it was quickly reverted because it was not well discussed and contradicts the documentation.
msg347325 - (view) Author: Carl Friedrich Bolz-Tereick (Carl.Friedrich.Bolz) * Date: 2019-07-05 10:02
OK, fair enough. That means right now there is no way to find the position of the operator using the ast module at the moment, correct?
msg347326 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-07-05 10:11
It is between the end position of the left operand and the start position of the right operand.
msg347370 - (view) Author: Carl Friedrich Bolz-Tereick (Carl.Friedrich.Bolz) * Date: 2019-07-05 19:32
I think the attached patch fixes the problem. Will create a pull request soon.
msg347509 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2019-07-08 21:17
New changeset 110a47c4f42cf4db88edc1876899fff8f05190fb by Ivan Levkivskyi (Carl Friedrich Bolz-Tereick) in branch 'master':
bpo-18374: fix wrong col_offset of some ast.BinOp instances (GH-14607)
https://github.com/python/cpython/commit/110a47c4f42cf4db88edc1876899fff8f05190fb
msg347510 - (view) Author: miss-islington (miss-islington) Date: 2019-07-08 21:41
New changeset c7be35c2abd598f02a633879133caec356593241 by Miss Islington (bot) in branch '3.8':
bpo-18374: fix wrong col_offset of some ast.BinOp instances (GH-14607)
https://github.com/python/cpython/commit/c7be35c2abd598f02a633879133caec356593241
msg347554 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2019-07-09 12:20
New changeset 430a9f44fe22f029ae8cfeecb46621d7e199414b by Ivan Levkivskyi (Carl Friedrich Bolz-Tereick) in branch 'master':
bpo-18374: fix tests to check the correct thing about line numbers (GH-14659)
https://github.com/python/cpython/commit/430a9f44fe22f029ae8cfeecb46621d7e199414b
msg347560 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2019-07-09 13:29
New changeset 68bd9c5691c4899d21cc7fe6cce7cd22b2f5ccb0 by Ivan Levkivskyi (Miss Islington (bot)) in branch '3.8':
bpo-18374: fix tests to check the correct thing about line numbers (GH-14659) (GH-14672)
https://github.com/python/cpython/commit/68bd9c5691c4899d21cc7fe6cce7cd22b2f5ccb0
History
Date User Action Args
2019-07-09 19:37:39levkivskyisetstatus: open -> closed
stage: patch review -> resolved
resolution: fixed
versions: + Python 3.8, Python 3.9, - Python 3.5
2019-07-09 13:29:01levkivskyisetmessages: + msg347560
2019-07-09 12:39:53vstinnersetnosy: - vstinner
2019-07-09 12:20:13miss-islingtonsetpull_requests: + pull_request14480
2019-07-09 12:20:10levkivskyisetmessages: + msg347554
2019-07-09 09:15:38Carl.Friedrich.Bolzsetpull_requests: + pull_request14466
2019-07-08 21:41:37miss-islingtonsetnosy: + miss-islington
messages: + msg347510
2019-07-08 21:18:11miss-islingtonsetpull_requests: + pull_request14460
2019-07-08 21:17:59levkivskyisetmessages: + msg347509
2019-07-05 20:46:32Carl.Friedrich.Bolzsetstage: needs patch -> patch review
pull_requests: + pull_request14423
2019-07-05 19:32:26Carl.Friedrich.Bolzsetfiles: + binop-offset.patch

messages: + msg347370
versions: + Python 3.5, - Python 3.7, Python 3.8, Python 3.9
2019-07-05 10:11:20serhiy.storchakasetmessages: + msg347326
2019-07-05 10:02:03Carl.Friedrich.Bolzsetmessages: + msg347325
2019-07-05 09:51:37serhiy.storchakasetmessages: + msg347324
2019-07-05 09:38:36Aivar.Annamaasetmessages: + msg347320
2019-07-05 09:15:30vstinnersetnosy: + vstinner
messages: + msg347319
2019-07-05 09:14:33vstinnersetnosy: + levkivskyi, pablogsal
2019-07-05 08:46:19serhiy.storchakasetversions: + Python 3.7, Python 3.8, Python 3.9, - Python 3.5
nosy: + serhiy.storchaka

messages: + msg347313

assignee: serhiy.storchaka
stage: patch review -> needs patch
2019-07-04 20:15:34Carl.Friedrich.Bolzsetnosy: + Carl.Friedrich.Bolz
messages: + msg347294
2014-08-22 10:40:14Claudiu.Popasetstage: patch review
type: behavior
versions: + Python 3.5, - Python 3.2, Python 3.3
2014-04-17 15:19:52sam.kimbrelsetfiles: + 18374-binop-col-offset-with-test.diff

nosy: + sam.kimbrel
messages: + msg216696

keywords: + patch
2013-07-12 22:18:38terry.reedysetnosy: + benjamin.peterson
2013-07-06 09:35:02Aivar.Annamaacreate