classification
Title: Minor inaccuracy in shlex.shlex punctuation_chars example
Type: behavior Stage: resolved
Components: Documentation Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: berker.peksag, docs@python, evan_, marco.buttu, python-dev, r.david.murray, vinay.sajip
Priority: normal Keywords: patch

Created on 2017-01-02 13:00 by evan_, last changed 2017-01-09 16:48 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
issue29133.patch marco.buttu, 2017-01-02 17:06
issue29133_2nd.patch marco.buttu, 2017-01-03 16:52
issue29133_3rd.patch marco.buttu, 2017-01-09 11:19
Messages (10)
msg284481 - (view) Author: Evan Andrews (evan_) * Date: 2017-01-02 13:00
https://docs.python.org/3/library/shlex.html#improved-compatibility-with-shells

The code sample here does not match the output - the first line is labelled 'New' and the second line 'Old'.
msg284490 - (view) Author: Marco Buttu (marco.buttu) * Date: 2017-01-02 17:06
Thank you. The patch fixes it and makes the example under doctest.
msg284531 - (view) Author: Evan Andrews (evan_) * Date: 2017-01-03 02:18
Sorry for not being more clear in the original report - the error is in the code, not in the output. The old behavior is punct=False, the new is punct=True.
msg284582 - (view) Author: Marco Buttu (marco.buttu) * Date: 2017-01-03 16:52
Yes, you are right. Here is a patch.
msg284599 - (view) Author: Evan Andrews (evan_) * Date: 2017-01-03 22:58
Second patch looks good, thanks. Do you also want to doctest that?
msg284622 - (view) Author: Marco Buttu (marco.buttu) * Date: 2017-01-04 09:14
I did not add the doctest directive on purpose. In fact, if in the future we start running the doctests with buildbot (see issue27200), the tests related to this issue will pass only for Python >= 3.6. If we _currently_ want the doctests to pass for every Python version, I see only two solutions:

1) we do not have to doctest the example, as I did in issue29133_2nd.patch
2) we have to apply two patches, one with doctests (for the doc to be tested with Python >= 3.6), and another one without doctests (for the doc to be tested with Python < 3.6).

I think the best solution is to add an option to the doctest directive, that allows us to specify for which Python version the tests have to be run:

https://github.com/sphinx-doc/sphinx/issues/3303

I would like the commiters to give a feedback about this, because if they think  this solution is doable, then I will spend the time to work on the Sphinx issue.
msg284894 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2017-01-07 06:49
I'd probably write it without the for loop:

    text = "a && b; c && d || e; f >'abc'; (def \"ghi\")"

    result = shlex.shlex(text)
    print(f"Old behavior: {list(result)}")

    result = shlex.shlex(text, punctuation_chars=True)
    print(f"New behavior: {list(result)}")

Or just:

    >>> import shlex
    >>> text = "a && b; c && d || e; f >'abc'; (def \"ghi\")"
    >>> list(shlex.shlex(text))
    ['a', '&', '&', 'b', ';', 'c', '&', '&', 'd', '|', '|', 'e', ';', 'f', '>', "'abc'", ';', '(', 'def', '"ghi"', ')']
    >>> list(shlex.shlex(text, punctuation_chars=True))
    ['a', '&&', 'b', ';', 'c', '&&', 'd', '||', 'e', ';', 'f', '>', "'abc'", ';', '(', 'def', '"ghi"', ')']

(Adding Vinay to nosy list to get his feedback since he wrote the original example.)
msg284906 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2017-01-07 11:31
I'm fine with removing the loop, as there are only ever going to be the two cases - so Berker's suggestion would seem to cover both doctest and unittest cases.
msg285036 - (view) Author: Marco Buttu (marco.buttu) * Date: 2017-01-09 11:19
Here is a 3rd patch with the Berker's suggestion. I just limited the output to 79 characters per line, and made use of the +NORMALIZE_WHITESPACE option  (to normalize the newline inside the output).
msg285052 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2017-01-09 16:48
New changeset af5b34b2d169 by Vinay Sajip in branch '3.6':
Fixes #29133: clarified shlex documentation.
https://hg.python.org/cpython/rev/af5b34b2d169

New changeset e3d820c0c884 by Vinay Sajip in branch 'default':
Closes #29133: merged update from 3.6.
https://hg.python.org/cpython/rev/e3d820c0c884
History
Date User Action Args
2017-01-09 16:48:44python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg285052

resolution: fixed
stage: patch review -> resolved
2017-01-09 11:19:05marco.buttusetfiles: + issue29133_3rd.patch

messages: + msg285036
2017-01-07 11:31:27vinay.sajipsetmessages: + msg284906
2017-01-07 06:49:43berker.peksagsetnosy: + berker.peksag, vinay.sajip
messages: + msg284894

type: behavior
stage: patch review
2017-01-04 09:27:25marco.buttusetnosy: + r.david.murray
2017-01-04 09:14:58marco.buttusetmessages: + msg284622
2017-01-03 22:58:20evan_setmessages: + msg284599
2017-01-03 16:52:28marco.buttusetfiles: + issue29133_2nd.patch

messages: + msg284582
2017-01-03 02:18:08evan_setmessages: + msg284531
2017-01-02 17:06:09marco.buttusetfiles: + issue29133.patch

nosy: + marco.buttu
messages: + msg284490

keywords: + patch
2017-01-02 13:00:12evan_create