classification
Title: shlex with string ending in space gives "ValueError: No closing quotation"
Type: behavior Stage: needs patch
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: ekorn, meador.inge, r.david.murray
Priority: normal Keywords:

Created on 2011-12-06 22:33 by ekorn, last changed 2011-12-09 22:06 by r.david.murray. This issue is now closed.

Messages (5)
msg148941 - (view) Author: (ekorn) Date: 2011-12-06 22:33
It seems shlex fails on processing strings ending in space, causing this bug that I reported for IPython:
https://github.com/ipython/ipython/issues/1109

Fernando Perez made a minimal example of the problem, quoted below. As he points out, it would be best to fix this at the source, namely the shlex module.

Python 2.7.2 |EPD 7.1-1 (32-bit)| (default, Jul  3 2011, 15:13:59) [MSC v.1500 32 bit (Intel)] on win32
>>> import shlex
>>> lex = shlex.shlex(' ("a ")', posix=False)
>>> lex.whitespace_split = True
>>> list(lex)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\shlex.py", line 269, in next
    token = self.get_token()
  File "C:\Python27\lib\shlex.py", line 96, in get_token
    raw = self.read_token()
  File "C:\Python27\lib\shlex.py", line 172, in read_token
    raise ValueError, "No closing quotation"
ValueError: No closing quotation
msg148942 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-12-06 22:55
Why do you consider this to be a bug?  You set posix=False and whitespace_split=True, so it seems to me that according to the documented rules the " inside the word ("a is as documented not recognized as a quote character.  If you use either posix=True or whitespace_split=False the string parses without error.  (Since I have no idea what standard non-posix mode is based on, I have no idea whether or not this is in fact correct behavior, though.)
msg149061 - (view) Author: (ekorn) Date: 2011-12-08 23:17
FYI, Min RK commented on the IPython issue:
https://github.com/ipython/ipython/issues/1109#issuecomment-3071470

In short, 
shlex.shlex('blob f(" ")', posix=False)
fails, whereas 
shlex.shlex('blob f( " ")', posix=False)

"The problem appears to be that Python source obviously doesn't sit well with non-posix whitespace-split shlex. [...] if you lead all your open-quotes with whitespace, you should be fine (works for all given examples, at least). [...] I have no idea whether this is a Python bug or not, since I don't know what the reference standard is, but this is definitely an IPython bug.  We should not be trying to use shlex to parse Python code as if it were command-line arguments."
msg149129 - (view) Author: (ekorn) Date: 2011-12-09 21:46
https://github.com/ipython/ipython/issues/1109#issuecomment-3072571
It seems this was an IPython bug due to slight abuse of the shlex module. Closing as invalid; thanks for your help.
msg149131 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-12-09 22:06
And just for your information, as far as I know *no one* knows what standard (or model) non-posix mode shlex is based on.
History
Date User Action Args
2011-12-09 22:06:07r.david.murraysetmessages: + msg149131
2011-12-09 21:46:15ekornsetstatus: open -> closed
resolution: not a bug
messages: + msg149129
2011-12-08 23:17:07ekornsetmessages: + msg149061
2011-12-06 22:55:54r.david.murraysetnosy: + r.david.murray
messages: + msg148942
2011-12-06 22:53:14meador.ingesetnosy: + meador.inge

stage: needs patch
2011-12-06 22:33:53ekorncreate