classification
Title: Exception from 'compile' reports a newline char not present in input
Type: behavior Stage: patch review
Components: Interpreter Core Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Anthony Sottile, BCSharp, SilentGhost, benjamin.peterson, brett.cannon, gvanrossum, yselivanov
Priority: normal Keywords: patch

Created on 2019-05-14 20:28 by BCSharp, last changed 2019-05-28 22:28 by BCSharp.

Pull Requests
URL Status Linked Edit
PR 13639 open BCSharp, 2019-05-28 22:28
Messages (7)
msg342515 - (view) Author: Pavel Koneski (BCSharp) * Date: 2019-05-14 20:28
Since Python 3.2, input in 'exec' mode  of 'compile' does not have to end in a newline anymore. However, it creates a surprising behavior when a 'SyntaxError' is reported:

>>> try: compile('try', '<string>', 'exec')
... except SyntaxError as ex: print(repr(ex))
... 
SyntaxError('invalid syntax', ('<string>', 1, 4, 'try\n'))

The 'text' field of the exception thrown contains an additional newline character that was not present in the input. Is it:

a. Proper Python language behavior?
b. CPython implementation artifact?
c. A bug?

In case of:

a. I will submit a patch to IronPython, which does not add an extra newline at the moment.
b. I can submit a patch to CPython to make StdLib tests implementation independent.
c. This inquiry can serve as a bug report.
msg342552 - (view) Author: SilentGhost (SilentGhost) * (Python triager) Date: 2019-05-15 07:47
I don't think a roundtrip is guaranteed by the parser, only producing an equivalent input.
msg342586 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-05-15 16:57
I don't think it's option C. As for whether A or B I don't know. Wouldn't hurt to ask on python-dev if no one comes forward with a more authoritative answer.
msg342831 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2019-05-19 03:30
still not able to answer the why, but at least I can answer the what here:

https://github.com/python/cpython/blob/f665b96e92a6a6943e312e2c606f348db95939ab/Parser/tokenizer.c#L984-L987

cpython adds a newline during tokenization if the file does not end in a newline
msg342944 - (view) Author: Pavel Koneski (BCSharp) * Date: 2019-05-20 19:04
If "equivalent input" is acceptable, then it looks like case B: other implementations possibly having different forms of equivalent input.

I am going to post this question on python-dev.
msg342947 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-05-20 19:45
I'm assuming the real issue is wanting to make IronPython pass as much of the CPython test suite as possible. I am okay with interpretation (B) in this case -- I can totally see that other parsing strategies have no use for adding the '\n' character to the end of the string. But I don't want to declare that CPython is wrong to show the '\n' -- it seems a pretty harmless artifact. So fixing the tests to allow it either way sounds good.
msg342950 - (view) Author: Pavel Koneski (BCSharp) * Date: 2019-05-20 20:27
> I'm assuming the real issue is wanting to make IronPython pass as much of the CPython test suite as possible.

This is indeed the case. The CPython test suite is invaluable in guiding IronPython development. Most of the time, the tests are pretty good to gloss over implementation artifacts (usually error messages), so that they work for IronPython as well, despite some differences between CPython and IronPython.

There are a few cases, however, when the tests expect behavior that is implementation-specific and difficult to match in IronPython, or is proper Python but impossible to match for IronPython. For all such cases I would like to submit patches to the CPython repo, but I am new to this process. Should such case first be reported on bpo, python-dev, or just straight a github PR?

Sometimes, by writing additional tests for IronPython we discover what seems as possible bugs in CPython. I was planning to submit reports for them on bpo, assuming this is the proper place to discuss them, but perhaps python-dev is a better place.

> So fixing the tests to allow it either way sounds good.

A github PR is on its way.
History
Date User Action Args
2019-05-28 22:28:53BCSharpsetkeywords: + patch
stage: patch review
pull_requests: + pull_request13535
2019-05-20 20:27:20BCSharpsetmessages: + msg342950
2019-05-20 19:45:58gvanrossumsetnosy: + gvanrossum
messages: + msg342947
2019-05-20 19:05:47BCSharpsetversions: - Python 3.5, Python 3.6
2019-05-20 19:04:41BCSharpsetmessages: + msg342944
versions: + Python 3.5, Python 3.6
2019-05-19 03:30:27Anthony Sottilesetnosy: + Anthony Sottile
messages: + msg342831
2019-05-15 16:57:33brett.cannonsetmessages: + msg342586
2019-05-15 07:47:12SilentGhostsetnosy: + SilentGhost, brett.cannon, yselivanov, benjamin.peterson
title: Exception form 'compile' reports a newline char not present in input -> Exception from 'compile' reports a newline char not present in input
messages: + msg342552

versions: - Python 3.5, Python 3.6
2019-05-14 20:28:37BCSharpcreate