classification
Title: Bytes version of sys.argv
Type: enhancement Stage: needs patch
Components: Interpreter Core, Unicode Versions: Python 3.5
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: ncoghlan Nosy List: Arfrever, amaury.forgeotdarc, ezio.melotti, lemburg, loewis, ncoghlan, rhettinger, vstinner
Priority: normal Keywords:

Created on 2010-05-20 12:10 by vstinner, last changed 2014-04-29 06:28 by ncoghlan. This issue is now closed.

Files
File name Uploaded Description Edit
argvb.py vstinner, 2010-10-24 20:35
Messages (16)
msg106140 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-05-20 12:10
In some situations, the encoding of the command line is incorrect or unknown. sys.argv is decoded with the file system encoding which can be wrong. Eg. see issue #4388 (ok, it's a bug, it should be fixed).

As os.environb, it would be useful to have bytes version of sys.argv to have able to decide the encoding used to decode each argument, or to manipulate bytes if we don't care about the encoding.

See also issue #8775 which propose to add a new encoding to decode sys.argv.
msg106141 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-05-20 12:29
> sys.argv is decoded with the file system encoding

IIRC this is not exact. Py_Main signature is 
    Py_Main(int argc, wchar_t **argv)
then PyUnicode_FromWideChar is used, and there is no conversion (except from UCS4 to UCS2).
The wchar_t strings themselves are built with mbstowcs(), the file system encoding is not used.
msg106143 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-05-20 12:39
> The wchar_t strings themselves are built with mbstowcs(), 
> the file system encoding is not used.

Oops sorry, you are right, and it's worse :-) sys.argv is decoded using the locale encoding, but subprocess & cie use the file system encoding for the reverse operation. => it doesn't work if both encodings are different (#4388, #8775).

The pseudo-code to create sys.argv on Unix is:

 # argv is a bytes list
 encoding = locale.getpreferredencoding()
 sys.argv = [arg.decode(encoding, 'surrogateescape') for arg in argv]
msg106172 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-05-20 17:24
> As os.environb, it would be useful to have bytes version of sys.argv
> to have able to decide the encoding used to decode each argument, or
> to manipulate bytes if we don't care about the encoding.

-1. Py_Main expects wchar_t*, so no byte-oriented representation of the
command line is readily available.
msg111754 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-07-28 00:53
"no byte-oriented representation of the command line is readily available."

Why not using the following recipe?

 encoding = locale.getpreferredencoding()
 sys.argvb = [arg.decode(encoding, 'surrogateescape') for arg in argv]
msg111757 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-07-28 01:21
You should read .encode(), not .decode() :-/
msg111770 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-07-28 05:54
Using that approach would work on POSIX systems.

Another problem I see is synchronizing the two. If some function strips arguments from sys.argv (because it has completed processing), sys.argvb would still keep the arguments. Of course, this could be fixed by having sys.argvb be a dynamic list (i.e. a sequence object) instead.
msg111818 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-07-28 14:50
> Using that approach would work on POSIX systems.

As os.environb, I think that sys.argv should not exist on Windows.

> Another problem I see is synchronizing the two

os.environ and os.environb are synchronized. It would be possible to do the same with sys.argv and sys.argvb. The implement would be simplier because it's just a list, not a dict.
msg111819 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-07-28 14:54
STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
>> Using that approach would work on POSIX systems.
> 
> As os.environb, I think that sys.argv should not exist on Windows.
> 
>> Another problem I see is synchronizing the two
> 
> os.environ and os.environb are synchronized. It would be possible to do the same with sys.argv and sys.argvb. The implement would be simplier because it's just a list, not a dict.

+1 on adding sys.argvb for systems that use char* in main().
msg119255 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-10-21 00:58
Since r85765 (issue #4388), always use UTF-8 to decode the command line arguments on Mac OS X, not the locale encoding. Which means that the pseudo-code becomes:

 if os.name != 'nt':
     if sys.platform == 'darwin':
        encoding = 'utf-8'
     else:
        encoding = locale.getpreferredencoding()
     sys.argvb = [arg.decode(encoding, 'surrogateescape') for arg in sys.argv]

sys.argvb should be synchronized with sys.argv, as os.environb with os.environ.
msg119528 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-10-24 20:35
Prototype (in Python) of argvb.py. Try it with: ./python -i argvb.py.

It's not possible to create sys.argvb in Python in a module loaded by Py_Initialize(), because sys.argv is created after Py_Initialize().
msg133608 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-04-12 22:25
One year after opening the issue, I don't have any real use case. And there are technical issues to implement this feature, so I prefer just to close this issue. Reopen it if you really want it, but please give an use case ;-)
msg217377 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-04-28 15:11
I'd like to revisit this after PEP 432 is in place, since having to do this dance for arg processing when running on Linux in the POSIX locale is somewhat lame:

    argv = sys.argv
    encoding = locale.getpreferredencoding() # Hope nobody changed the locale!
    fixed_encoding = read_encoding_from("/etc/locale.conf") # For example
    argvb = [arg.encode(encoding, "surrogateescape") for arg in argv]
    fixed_argv = [arg.decode(fixed_encoding, "surrogateescape") for arg in argvb]

(For stricter parsing, leave out the second "surrogateescape")

Now, if PEP 432 resolves the system encoding issue such that we are able to use the right encoding even when locale.getpreferredencoding() returns the wrong answer, then it may not be worthwhile to also provide sys.argvb (especially since it won't help hybrid 2/3 code). On the other hand, like os.environb, it does make it easier for POSIX-only code paths that wants to handle boundary encoding issues directly to stick with consuming the binary data directly and avoid the interpreter's automatic conversion to the text domain.

Note also that os.environb is only available when os.supports_bytes_environ is True, so it would make sense to only provide sys.argvb in the circumstances where we provide os.environb.
msg217408 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2014-04-28 19:51
Without commenting on this specific proposal, I would like to make an overall observation that Python is impairing its usability by adding too-many-ways-to-it in a number of categories (file descriptor variants of file methods, multiple versions of time.time, byte variants of everything that is done with strings).  Python 3 was intended to be a cleaner, more learnable version of Python.  Instead, it is growing enums, multiple dispatch, and multiple variants of every function.   Professional programmers can be well served by some of the these tools, but the Python universe is much larger than that and the other users are not being well served by these additions (too many choices impairs usability and learnability).
msg217416 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-04-28 20:49
Today I regret os.environb (I added it). If I remember correctly, os.environb was added before the PEP 383 (surrogateescape). This PEP makes os.environb almost useless. In Python 3, Unicode is the natural choice, and thanks to the PEP 383, it's still possible to use any "raw bytes".

argvb can be computed in one line: list(map(os.fsencode, sys.argv)).

I now suggest to close this issue as wontfix.
msg217475 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-04-29 06:28
Makes sense to me. Assuming we eventually manage to resolve the POSIX locale issue, the bytes variant will become even less useful.
History
Date User Action Args
2014-04-29 06:28:30ncoghlansetstatus: open -> closed
resolution: later -> rejected
messages: + msg217475
2014-04-28 20:49:24vstinnersetmessages: + msg217416
2014-04-28 19:51:23rhettingersetnosy: + rhettinger
messages: + msg217408
2014-04-28 15:11:54ncoghlansetstatus: closed -> open

assignee: ncoghlan
versions: + Python 3.5, - Python 3.3
nosy: + ncoghlan

messages: + msg217377
resolution: wont fix -> later
2011-04-12 22:25:11vstinnersetstatus: open -> closed
resolution: wont fix
messages: + msg133608
2010-12-14 18:58:30r.david.murraysetstage: needs patch
type: enhancement
versions: + Python 3.3, - Python 3.2
2010-10-24 20:35:57vstinnersetfiles: + argvb.py

messages: + msg119528
2010-10-21 00:58:20vstinnersetmessages: + msg119255
2010-07-28 14:54:27lemburgsetnosy: + lemburg
messages: + msg111819
2010-07-28 14:50:31vstinnersetmessages: + msg111818
2010-07-28 05:54:44loewissetmessages: + msg111770
2010-07-28 03:42:31ezio.melottisetnosy: + ezio.melotti
2010-07-28 01:21:04vstinnersetmessages: + msg111757
2010-07-28 00:53:02vstinnersetmessages: + msg111754
2010-05-20 17:24:32loewissetnosy: + loewis
messages: + msg106172
2010-05-20 16:30:21Arfreversetnosy: + Arfrever
2010-05-20 12:39:43vstinnersetmessages: + msg106143
2010-05-20 12:29:33amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg106141
2010-05-20 12:10:54vstinnercreate