classification
Title: Exception from 'compile' reports a newline char not present in input
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Anthony Sottile, BCSharp, SilentGhost, benjamin.peterson, brett.cannon, gvanrossum, jaraco, yselivanov
Priority: normal Keywords: patch

Created on 2019-05-14 20:28 by BCSharp, last changed 2019-09-11 14:37 by jaraco. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 13639 merged BCSharp, 2019-05-28 22:28
PR 15952 merged miss-islington, 2019-09-11 13:57
PR 15953 merged miss-islington, 2019-09-11 13:57
Messages (10)
msg342515 - (view) Author: Pavel Koneski (BCSharp) * Date: 2019-05-14 20:28
Since Python 3.2, input in 'exec' mode  of 'compile' does not have to end in a newline anymore. However, it creates a surprising behavior when a 'SyntaxError' is reported:

>>> try: compile('try', '<string>', 'exec')
... except SyntaxError as ex: print(repr(ex))
... 
SyntaxError('invalid syntax', ('<string>', 1, 4, 'try\n'))

The 'text' field of the exception thrown contains an additional newline character that was not present in the input. Is it:

a. Proper Python language behavior?
b. CPython implementation artifact?
c. A bug?

In case of:

a. I will submit a patch to IronPython, which does not add an extra newline at the moment.
b. I can submit a patch to CPython to make StdLib tests implementation independent.
c. This inquiry can serve as a bug report.
msg342552 - (view) Author: SilentGhost (SilentGhost) * (Python triager) Date: 2019-05-15 07:47
I don't think a roundtrip is guaranteed by the parser, only producing an equivalent input.
msg342586 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-05-15 16:57
I don't think it's option C. As for whether A or B I don't know. Wouldn't hurt to ask on python-dev if no one comes forward with a more authoritative answer.
msg342831 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2019-05-19 03:30
still not able to answer the why, but at least I can answer the what here:

https://github.com/python/cpython/blob/f665b96e92a6a6943e312e2c606f348db95939ab/Parser/tokenizer.c#L984-L987

cpython adds a newline during tokenization if the file does not end in a newline
msg342944 - (view) Author: Pavel Koneski (BCSharp) * Date: 2019-05-20 19:04
If "equivalent input" is acceptable, then it looks like case B: other implementations possibly having different forms of equivalent input.

I am going to post this question on python-dev.
msg342947 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-05-20 19:45
I'm assuming the real issue is wanting to make IronPython pass as much of the CPython test suite as possible. I am okay with interpretation (B) in this case -- I can totally see that other parsing strategies have no use for adding the '\n' character to the end of the string. But I don't want to declare that CPython is wrong to show the '\n' -- it seems a pretty harmless artifact. So fixing the tests to allow it either way sounds good.
msg342950 - (view) Author: Pavel Koneski (BCSharp) * Date: 2019-05-20 20:27
> I'm assuming the real issue is wanting to make IronPython pass as much of the CPython test suite as possible.

This is indeed the case. The CPython test suite is invaluable in guiding IronPython development. Most of the time, the tests are pretty good to gloss over implementation artifacts (usually error messages), so that they work for IronPython as well, despite some differences between CPython and IronPython.

There are a few cases, however, when the tests expect behavior that is implementation-specific and difficult to match in IronPython, or is proper Python but impossible to match for IronPython. For all such cases I would like to submit patches to the CPython repo, but I am new to this process. Should such case first be reported on bpo, python-dev, or just straight a github PR?

Sometimes, by writing additional tests for IronPython we discover what seems as possible bugs in CPython. I was planning to submit reports for them on bpo, assuming this is the proper place to discuss them, but perhaps python-dev is a better place.

> So fixing the tests to allow it either way sounds good.

A github PR is on its way.
msg351883 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2019-09-11 13:57
New changeset b6643dcfc26859f935e4b3a6a2a203e8ef5320e2 by Jason R. Coombs (Pavel Koneski) in branch 'master':
bpo-36919: make test_source_encoding.test_issue2301 implementation-independent (#13639)
https://github.com/python/cpython/commit/b6643dcfc26859f935e4b3a6a2a203e8ef5320e2
msg351905 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2019-09-11 14:36
New changeset c2119399488a297275db7f0590cff957b6ce8a2e by Jason R. Coombs (Miss Islington (bot)) in branch '3.7':
bpo-36919: make test_source_encoding.test_issue2301 implementation-independent (GH-13639) (GH-15953)
https://github.com/python/cpython/commit/c2119399488a297275db7f0590cff957b6ce8a2e
msg351907 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2019-09-11 14:37
New changeset 42edfcfd129c7ac4842b9e0cd453bbe3ff006669 by Jason R. Coombs (Miss Islington (bot)) in branch '3.8':
bpo-36919: make test_source_encoding.test_issue2301 implementation-independent (GH-13639) (GH-15952)
https://github.com/python/cpython/commit/42edfcfd129c7ac4842b9e0cd453bbe3ff006669
History
Date User Action Args
2019-09-11 14:37:37jaracosetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-09-11 14:37:21jaracosetmessages: + msg351907
2019-09-11 14:36:38jaracosetmessages: + msg351905
2019-09-11 13:57:50miss-islingtonsetpull_requests: + pull_request15588
2019-09-11 13:57:44miss-islingtonsetpull_requests: + pull_request15587
2019-09-11 13:57:35jaracosetnosy: + jaraco
messages: + msg351883
2019-09-11 13:56:19jaracosetversions: + Python 3.9
2019-05-28 22:28:53BCSharpsetkeywords: + patch
stage: patch review
pull_requests: + pull_request13535
2019-05-20 20:27:20BCSharpsetmessages: + msg342950
2019-05-20 19:45:58gvanrossumsetnosy: + gvanrossum
messages: + msg342947
2019-05-20 19:05:47BCSharpsetversions: - Python 3.5, Python 3.6
2019-05-20 19:04:41BCSharpsetmessages: + msg342944
versions: + Python 3.5, Python 3.6
2019-05-19 03:30:27Anthony Sottilesetnosy: + Anthony Sottile
messages: + msg342831
2019-05-15 16:57:33brett.cannonsetmessages: + msg342586
2019-05-15 07:47:12SilentGhostsetnosy: + SilentGhost, brett.cannon, yselivanov, benjamin.peterson
title: Exception form 'compile' reports a newline char not present in input -> Exception from 'compile' reports a newline char not present in input
messages: + msg342552

versions: - Python 3.5, Python 3.6
2019-05-14 20:28:37BCSharpcreate