classification
Title: Obscure netrc parser "bug"
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: bbayles, skip.montanaro, xiang.zhang, xtreak
Priority: normal Keywords: patch

Created on 2018-07-16 23:13 by skip.montanaro, last changed 2018-07-21 00:54 by bbayles.

Files
File name Uploaded Description Edit
netrc-comment-blank skip.montanaro, 2018-07-16 23:13 working netrc file
netrc-blank-comment skip.montanaro, 2018-07-16 23:13 problematic netrc file
netrc_foo_tester.py xtreak, 2018-07-17 10:14
Pull Requests
URL Status Linked Edit
PR 8320 closed bbayles, 2018-07-18 02:21
PR 8360 closed bbayles, 2018-07-21 00:52
Messages (4)
msg321779 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2018-07-16 23:13
Not sure I can really call this a bug, however there is a behavioral change between 2.7 and at least 3.6 and 3.7 (probably earlier versions of the 3.x series as well). There is no spec for .netrc files that I can find, certainly nothing which mentions comment or blank lines. Still, Python's netrc file parser seems happy with both.

However, in 3.x a blank line followed immediately by a comment line containing actual comment text causes the parser to raise a parse error. I've attached two netrc files, netrc-comment-blank, and netrc-blank-comment, identical save for the ordering of a blank line and a comment line. Here's what a 2.7.14 session looks like:

Python 2.7.14 |Anaconda, Inc.| (default, Mar 27 2018, 17:29:31) 
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import netrc
>>> rc = netrc.netrc(file="/home/skip/tmp/netrc-comment-blank")
>>> rc = netrc.netrc(file="/home/skip/tmp/netrc-blank-comment")

Here's 3.7.0:

Python 3.7.0 (default, Jun 28 2018, 13:15:42) 
[GCC 7.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import netrc
>>> rc = netrc.netrc(file="/home/skip/tmp/netrc-comment-blank")
>>> rc = netrc.netrc(file="/home/skip/tmp/netrc-blank-comment")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/skip/miniconda3/envs/python3/lib/python3.7/netrc.py", line 30, in __init__
    self._parse(file, fp, default_netrc)
  File "/home/skip/miniconda3/envs/python3/lib/python3.7/netrc.py", line 63, in _parse
    "bad toplevel token %r" % tt, file, lexer.lineno)
netrc.NetrcParseError: bad toplevel token 'Comment' (/home/skip/tmp/netrc-blank-comment, line 2)
msg321810 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2018-07-17 10:14
This seems to have been introduced with commit : 1df0f214a9fdb4dde7506576b144cf6a7fd01b65 before which it was working fine.

Testing shows the error from 1df0f214a9fdb4dde7506576b144cf6a7fd01b65 and above. I have checked out only versions of Lib/netrc.py and I think this doesn't require a rebuild of Python.

➜  cpython git:(master) $ git log --oneline --format="%h" Lib/netrc.py > commits.txt # Get all commits to test the changes
➜  cpython git:(master) ✗ ./python netrc_foo_tester.py

netrc_foo_tester.py is attached with the message which basically does the below using Python for every commit. netrc_foo.py contains the actual test of parsing two files.

➜  cpython git:(master) ✗ ./python
Python 3.8.0a0 (heads/master:35c0809, Jul 16 2018, 10:29:23)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

➜ cpython git:(master) ✗ cat netrc_foo.py
import netrc

rc = netrc.netrc(file="netrc-comment-blank")
rc = netrc.netrc(file="netrc-blank-comment")

➜  cpython git:(master) ✗ git reset --quiet HEAD . && git checkout . && git checkout be19ed7 Lib/netrc.py && ./python netrc_foo.py
netrc_foo.py:2: ResourceWarning: unclosed file <_io.TextIOWrapper name='netrc-comment-blank' mode='r' encoding='UTF-8'>
  rc = netrc.netrc(file="netrc-comment-blank")
netrc_foo.py:3: ResourceWarning: unclosed file <_io.TextIOWrapper name='netrc-blank-comment' mode='r' encoding='UTF-8'>
  rc = netrc.netrc(file="netrc-blank-comment")
➜  cpython git:(master) ✗ git reset --quiet HEAD . && git checkout . && git checkout c12a813 Lib/netrc.py && ./python netrc_foo.py
➜  cpython git:(master) ✗ git reset --quiet HEAD . && git checkout . && git checkout 78a1a15 Lib/netrc.py && ./python netrc_foo.py
➜  cpython git:(master) ✗ git reset --quiet HEAD . && git checkout . && git checkout 1df0f21 Lib/netrc.py && ./python netrc_foo.py
Traceback (most recent call last):
  File "netrc_foo.py", line 3, in <module>
    rc = netrc.netrc(file="netrc-blank-comment")
  File "/home/cpython/Lib/netrc.py", line 32, in __init__
    self._parse(file, fp)
  File "/home/cpython/Lib/netrc.py", line 65, in _parse
    "bad toplevel token %r" % tt, file, lexer.lineno)
netrc.NetrcParseError: bad toplevel token 'Comment' (netrc-blank-comment, line 2)

I hope the above approach is correct.

Thanks
msg321975 - (view) Author: bbayles (bbayles) * Date: 2018-07-20 03:12
I took a shot at this, but closed my PR when I found that it introduced a different regression.

After playing with it for a bit, I think that this commit [1] is a bit problematic because it uses lexer.instream.readline(), which can make the line number in lexer.lineno incorrect.

I may take another look if I have time, but someone else is free to tackle this.

[1] https://github.com/python/cpython/commit/1df0f214a9fdb4dde7506576b144cf6a7fd01b65#diff-e6896f6d68ca3fd8094a933533f8b2ed
msg322053 - (view) Author: bbayles (bbayles) * Date: 2018-07-21 00:54
I realized that the Python 2 solution was adapt-able after all; I've re-submitted as GitHub PR 8360.

https://github.com/python/cpython/pull/8360
History
Date User Action Args
2018-07-21 00:54:36bbaylessetmessages: + msg322053
2018-07-21 00:52:50bbaylessetpull_requests: + pull_request7895
2018-07-20 03:12:47bbaylessetmessages: + msg321975
2018-07-18 02:21:22bbaylessetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request7856
2018-07-17 10:14:36xtreaksetfiles: + netrc_foo_tester.py

messages: + msg321810
2018-07-17 09:03:59xtreaksetnosy: + xtreak
2018-07-17 03:32:54xiang.zhangsetnosy: + xiang.zhang
2018-07-17 03:08:38bbaylessetnosy: + bbayles
2018-07-16 23:13:58skip.montanarosetfiles: + netrc-blank-comment
2018-07-16 23:13:33skip.montanarocreate