The shlex module does not function as expected in the presence of
comments when newlines are not whitespace. An example (attached):

>>> from shlex import shlex
>>> 
>>> lexer = shlex("a \n b")
>>> print ",".join(lexer)
a,b
>>> 
>>> lexer = shlex("a # comment \n b")
>>> print ",".join(lexer)
a,b
>>> 
>>> lexer = shlex("a \n b")
>>> lexer.whitespace=" "
>>> print ",".join(lexer)
a,
,b
>>> 
>>> lexer = shlex("a # comment \n b")
>>> lexer.whitespace=" "
>>> print ",".join(lexer)
a,b

Now where did my newline go? The comment ate it! Even though the docs
seem to indicate the newline is not part of the comment itself:

shlex.commenters:
    The string of characters that are recognized as comment beginners.
All characters from the comment beginner to end of line are ignored.
Includes just '#' by default.

Attached is a patch which fixes this for me. It basically does a
fall-through using '\n' when encountering a comment. So that may be a
bit of a hack (who says '\n' is the only newline char in there, and not
'\r'?) but I'll leave the more intricate stuff to you experts.

If you could add some tests to lib/test/test_shlex.py, there are more 
chances for this patch to be accepted.

Also, consider the case when the comment is on the last line of input 
and there is no \n ending character.