This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author ammar2
Recipients ammar2
Date 2018-06-19.07:41:51
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <>
As was pointed out in there is an edge case in the tokenizer whereby it will implicitly treat the end of input as a newline. The tokenize module in stdlib does not mirror the C code's behavior in this case.


  ~/cpython $ echo -n 'x' | ./python
  NAME ("x")

tokenize module:

  ~/cpython $ echo -n 'x' | ./python -m tokenize
  1,0-1,1:            NAME           'x'            
  2,0-2,0:            ENDMARKER      ''

The instrumentation to have the C tokenizer dump out its tokens is mine, can provide a diff to produce that output if needed.
Date User Action Args
2018-06-19 07:41:52ammar2setrecipients: + ammar2
2018-06-19 07:41:52ammar2setmessageid: <>
2018-06-19 07:41:52ammar2linkissue33899 messages
2018-06-19 07:41:51ammar2create