Title: Pre-feed the parser with the f-string expression location
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.10, Python 3.9
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: lys.nikolaou Nosy List: eric.smith, gvanrossum, lys.nikolaou, pablogsal
Priority: normal Keywords: patch

Created on 2020-06-22 12:54 by lys.nikolaou, last changed 2020-06-28 00:17 by lys.nikolaou. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 21054 merged lys.nikolaou, 2020-06-22 12:57
PR 21190 merged pablogsal, 2020-06-27 23:44
Messages (3)
msg372086 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-06-22 12:54
Inspired by bpo-41064, I sat down to try and find problems with f-string locations in the new parser. I was able to come up with a way to compute the locations of the f-string expressions that I think is more consistent and allows us to delete all the code that was fixing the expression locations after the actual parsing, which accounted for about 1/6 of string_parser.c.

A high-level explanation of the change:

Before this change we were pre-feeding the parser with the location of the f-string itself. The parser was then parsing the expression and was computing the locations of all the nodes based on the offset of the f-string. After the parsing was done, we were identifying the offset and the lineno of the expression *within* the fstring and were fixing the node locations accordingly. For example, for an f-string like `a = 0; f'irrelevant {a}'` we were doing the following:

- Pre-feed the parser with lineno=0 and col_offset=7 (the offset of the f-string itself in the current line).
- Parse the expression (adding 7 to the col_offset of each parsed node, lineno remains the same since it's 0).
- Fix the node locations by shifting the Name node by 14, which is the number of characters in the f-string (counting the `f` and the opening quote) before the start of the expression.

With this change we now pre-feed the parser with the exact lineno and offset of the expression itself, not the f-string. This allows us to completely skip the third step of shifting the node locations.
msg372481 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-27 23:41
New changeset 1f0f4abb110b9fbade6175842b6a26ab0b8df6dd by Lysandros Nikolaou in branch 'master':
bpo-41076: Pre-feed the parser with the f-string expression location (GH-21054)
msg372482 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-28 00:15
New changeset dab533d0ee067159812d4ea51f6fbbb1bd37d8b7 by Pablo Galindo in branch '3.9':
[3.9] bpo-41076: Pre-feed the parser with the f-string expression location (GH-21054) (GH-21190)
Date User Action Args
2020-06-28 00:17:18lys.nikolaousetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-06-28 00:15:35pablogsalsetmessages: + msg372482
2020-06-27 23:44:41pablogsalsetpull_requests: + pull_request20344
2020-06-27 23:41:52pablogsalsetmessages: + msg372481
2020-06-22 13:45:54lys.nikolaousettitle: Pre-feed the parser with the f-string location -> Pre-feed the parser with the f-string expression location
2020-06-22 12:57:41lys.nikolaousetkeywords: + patch
stage: patch review
pull_requests: + pull_request20225
2020-06-22 12:54:04lys.nikolaoucreate