When one of the items in the iterable passed to shlex.join() is a pathlib.Path object, it throws an exception saying it must be str or bytes. I believe it should accept Path objects just like other parts of the standard library such as subprocess.run() already do.


Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import shlex
>>> from pathlib import Path
>>> shlex.join(['foo', Path('bar baz')])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/shlex.py", line 320, in join
    return ' '.join(quote(arg) for arg in split_command)
  File "/usr/lib/python3.9/shlex.py", line 320, in <genexpr>
    return ' '.join(quote(arg) for arg in split_command)
  File "/usr/lib/python3.9/shlex.py", line 329, in quote
    if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object
>>> shlex.join(['foo', str(Path('bar baz'))])
"foo 'bar baz'"


Python 3.11.0a0 (heads/main:fa15df77f0, Sep  7 2021, 18:22:35) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import shlex
>>> from pathlib import Path
>>> shlex.join(['foo', Path('bar baz')])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nyuszika7h/.pyenv/versions/3.11-dev/lib/python3.11/shlex.py", line 320, in join
    return ' '.join(quote(arg) for arg in split_command)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nyuszika7h/.pyenv/versions/3.11-dev/lib/python3.11/shlex.py", line 320, in <genexpr>
    return ' '.join(quote(arg) for arg in split_command)
                    ^^^^^^^^^^
  File "/home/nyuszika7h/.pyenv/versions/3.11-dev/lib/python3.11/shlex.py", line 329, in quote
    if _find_unsafe(s) is None:
       ^^^^^^^^^^^^^^^
TypeError: expected string or bytes-like object, got 'PosixPath'
>>> shlex.join(['foo', str(Path('bar baz'))])
"foo 'bar baz'"

I disagree. In my mind, there's no more reason for shlex.join() to take Path objects than there is for str.join() to take them, or for shlex.join() to convert int's to str's. I'd rather shlex.join() continue to raise an exception if passed something that's not a str or bytes.

You should probably discuss this on the python-ideas mailing list if you want a broader audience.

While it may be primarily intended to combine output from shlex.split() again, IMO it's useful for manually constructed command lines as well, for example displaying instructions to a user where a path may contain spaces and special characters and needs to be properly escaped.

As for converting int to str, since subprocess.run() does not do that, shlex.split() does not need to do so either. I never mentioned that, and while I could see that being useful as well, that would have to be a separate discussion.

There's more of a case for automatic conversion for Path objects, which are supposed to work seamlessly in most places where strings are accepted. But such quirks of certain functions not accepting them and being forced to convert to str manually makes pathlib a little annoying to use compared to os.path.

My point is that shlex.join() shouldn’t convert any arguments to strings, no matter their type. Just like str.join() doesn’t, and for the same reason. 

Within the last few years there was a discussion on making str.join() auto-convert it’s arguments to str, and we decided not to, because it would mask too many bugs. I believe the same argument applies to shlex.join(). Path objects aren’t special enough to make an exception. 

There’s a proposal somewhere to also call str() on Path arguments to subprocess.run(). I’m opposed to that for the same reasons.

IMO comparing shlex.join() to str.join() is a mistake. Comparing it to subprocess.run() is more appropriate.

What do you mean by "proposal"? subprocess.run() already converts Path arguments to str since Python 3.6 (though IIRC this was broken on Windows until 3.7 or so). It indeed does not convert int arguments, but like I said that's irrelevant here, you're the one who brought it up.

I was using str.join() as an analogy. I apologize if that was confusing.

I'll state this as clearly as I can: I don't think shlex.join() should automatically convert any arguments to str, not even those that are Path's.

Nothing is going to be solved by having a discussion on the bug tracker. A discussion on python-ideas would be the appropriate place to take this. If a consensus is reached there to add this feature, then this feature request can be used for its implementation.

It's entirely possible people there might agree with the feature request, but I'd argue against it. One thing going for your proposal is subprocess.run(). I was thinking issue 38670 applied there, but I guess not.