This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: shlex punctuation_chars inconsistency
Type: behavior Stage: resolved
Components: Documentation Versions: Python 3.8, Python 3.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, rhettinger, tphh, vinay.sajip, xtreak
Priority: normal Keywords: easy, patch, patch, patch, patch

Created on 2018-11-05 17:32 by tphh, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 11631 merged python-dev, 2019-01-21 09:39
PR 15926 merged vinay.sajip, 2019-09-11 11:59
PR 15927 merged vinay.sajip, 2019-09-11 12:06
Messages (9)
msg329310 - (view) Author: (tphh) Date: 2018-11-05 17:32
The newly added shlex.punctuation_chars is special compared to the other public instance variables: It can ONLY be used when constructing a shlex instance, unlike other public instance variables, such as commenters, which can ONLY be set later.

>>> s = shlex.shlex('abc // def')
>>> s.commenters = '/'
>>> list(s)
['abc', '', '']

>>> s = shlex.shlex('abc // def', punctuation_chars = '/')
>>> list(s)
['abc', '//', 'def']

However, setting punctuation_chars later shows this rather useless error message:

>>> s = shlex.shlex('abc // def')
>>> s.punctuation_chars = '/'
>>> list(s)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/python/3.7.1/lib/python3.7/", line 295, in __next__
    token = self.get_token()
  File "/opt/python/3.7.1/lib/python3.7/", line 105, in get_token
    raw = self.read_token()
  File "/opt/python/3.7.1/lib/python3.7/", line 133, in read_token
    if self.punctuation_chars and self._pushback_chars:
AttributeError: 'shlex' object has no attribute '_pushback_chars'
msg329312 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2018-11-05 17:59
Thanks for the report. The code was added with c1f974c944a3e73cbc9102356d8700a190dcafb3 and self._pushback_chars is declared only when punctuation_chars is passed to shlex.shlex in the constructor as you have mentioned in . Adding Vinay for thoughts on the usage.
msg329315 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2018-11-05 19:45
I agree that it's inconsistent, but quite a bit of setting up is done when punctuation_chars is provided, as per the link in msg329312. One could convert the attribute to a property and have a setter that does the equivalent set up, but some of the setup is one-time (removing things from wordchars, etc.), and would require additional work to handle the case where the property is reassigned multiple times. I have no problem updating the documentation to indicate in a note that it must be provided in the constructor and not later, but apart from the fact that it's inconsistent, is there a use case for supporting setting it later? That would mean setting it up so that you could set it several times, unset it, etc.
msg329324 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2018-11-06 01:38
It makes sense to me that information used in an expensive one-time setup should be specified in advance where other parameters that are more easily changed are specified downstream.  The API reflects the a sensible way to use the tool.  Making it to easy to change later increases the risk of misuse.
msg329361 - (view) Author: (tphh) Date: 2018-11-06 14:39
So a documentation update and a better run time error message which clarifies that shlex.punctuation_chars is read-only?
msg329423 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2018-11-07 16:40
> a better run time error message which clarifies that shlex.punctuation_chars is read-only

That it can be set only via the __init__(), yes.
msg351820 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-09-11 11:04
New changeset 972cf5c06a5ba16ad243a442dbb9c15307fbed95 by Vinay Sajip (Alex) in branch 'master':
bpo-35168: Make shlex.punctuation_chars read-only (#11631)
msg351841 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-09-11 12:39
New changeset aca878ecf18b9e915e9b551f095a550b5c6e9bc5 by Vinay Sajip in branch '3.7':
[3.7] bpo-35168: Make shlex.punctuation_chars read-only (GH-11631) (GH-15926)
msg351842 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-09-11 12:39
New changeset 3b92ddb7612fd8fe2be95ff3d8022ed18335b13f by Vinay Sajip in branch '3.8':
[3.8] bpo-35168: Make shlex.punctuation_chars read-only (GH-11631) (GH-15927)
Date User Action Args
2022-04-11 14:59:07adminsetgithub: 79349
2019-09-11 12:47:41vinay.sajipsetkeywords: patch, patch, patch, patch, easy
status: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-09-11 12:39:55vinay.sajipsetmessages: + msg351842
2019-09-11 12:39:11vinay.sajipsetmessages: + msg351841
2019-09-11 12:06:42vinay.sajipsetpull_requests: + pull_request15567
2019-09-11 11:59:16vinay.sajipsetpull_requests: + pull_request15566
2019-09-11 11:04:07vinay.sajipsetmessages: + msg351820
2019-09-11 09:59:17vinay.sajipsetpull_requests: - pull_request11401
2019-09-11 09:59:07vinay.sajipsetpull_requests: - pull_request11402
2019-09-11 09:58:57vinay.sajipsetpull_requests: - pull_request11403
2019-01-21 09:40:09python-devsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request11403
2019-01-21 09:40:00python-devsetkeywords: + patch
stage: needs patch -> needs patch
pull_requests: + pull_request11402
2019-01-21 09:39:49python-devsetkeywords: + patch
stage: needs patch -> needs patch
pull_requests: + pull_request11401
2019-01-21 09:39:37python-devsetkeywords: + patch
stage: needs patch -> needs patch
pull_requests: + pull_request11400
2019-01-04 18:52:11cheryl.sabellasetnosy: + docs@python
versions: - Python 3.6
assignee: docs@python
components: + Documentation, - Library (Lib)
keywords: + easy
stage: needs patch
2018-11-07 16:40:57vinay.sajipsetmessages: + msg329423
2018-11-06 14:39:26tphhsetmessages: + msg329361
2018-11-06 01:38:25rhettingersetnosy: + rhettinger
messages: + msg329324
2018-11-05 19:45:07vinay.sajipsetmessages: + msg329315
2018-11-05 17:59:35xtreaksetnosy: + vinay.sajip, xtreak
messages: + msg329312
2018-11-05 17:32:14tphhcreate