Title: shlex punctuation_chars inconsistency
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: rhettinger, tphh, vinay.sajip, xtreak
Priority: normal Keywords:

Created on 2018-11-05 17:32 by tphh, last changed 2018-11-07 16:40 by vinay.sajip.

Messages (6)
msg329310 - (view) Author: (tphh) Date: 2018-11-05 17:32
The newly added shlex.punctuation_chars is special compared to the other public instance variables: It can ONLY be used when constructing a shlex instance, unlike other public instance variables, such as commenters, which can ONLY be set later.

>>> s = shlex.shlex('abc // def')
>>> s.commenters = '/'
>>> list(s)
['abc', '', '']

>>> s = shlex.shlex('abc // def', punctuation_chars = '/')
>>> list(s)
['abc', '//', 'def']

However, setting punctuation_chars later shows this rather useless error message:

>>> s = shlex.shlex('abc // def')
>>> s.punctuation_chars = '/'
>>> list(s)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/python/3.7.1/lib/python3.7/", line 295, in __next__
    token = self.get_token()
  File "/opt/python/3.7.1/lib/python3.7/", line 105, in get_token
    raw = self.read_token()
  File "/opt/python/3.7.1/lib/python3.7/", line 133, in read_token
    if self.punctuation_chars and self._pushback_chars:
AttributeError: 'shlex' object has no attribute '_pushback_chars'
msg329312 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2018-11-05 17:59
Thanks for the report. The code was added with c1f974c944a3e73cbc9102356d8700a190dcafb3 and self._pushback_chars is declared only when punctuation_chars is passed to shlex.shlex in the constructor as you have mentioned in . Adding Vinay for thoughts on the usage.
msg329315 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2018-11-05 19:45
I agree that it's inconsistent, but quite a bit of setting up is done when punctuation_chars is provided, as per the link in msg329312. One could convert the attribute to a property and have a setter that does the equivalent set up, but some of the setup is one-time (removing things from wordchars, etc.), and would require additional work to handle the case where the property is reassigned multiple times. I have no problem updating the documentation to indicate in a note that it must be provided in the constructor and not later, but apart from the fact that it's inconsistent, is there a use case for supporting setting it later? That would mean setting it up so that you could set it several times, unset it, etc.
msg329324 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2018-11-06 01:38
It makes sense to me that information used in an expensive one-time setup should be specified in advance where other parameters that are more easily changed are specified downstream.  The API reflects the a sensible way to use the tool.  Making it to easy to change later increases the risk of misuse.
msg329361 - (view) Author: (tphh) Date: 2018-11-06 14:39
So a documentation update and a better run time error message which clarifies that shlex.punctuation_chars is read-only?
msg329423 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2018-11-07 16:40
> a better run time error message which clarifies that shlex.punctuation_chars is read-only

That it can be set only via the __init__(), yes.
Date User Action Args
2018-11-07 16:40:57vinay.sajipsetmessages: + msg329423
2018-11-06 14:39:26tphhsetmessages: + msg329361
2018-11-06 01:38:25rhettingersetnosy: + rhettinger
messages: + msg329324
2018-11-05 19:45:07vinay.sajipsetmessages: + msg329315
2018-11-05 17:59:35xtreaksetnosy: + vinay.sajip, xtreak
messages: + msg329312
2018-11-05 17:32:14tphhcreate