In some situations, the encoding of the command line is incorrect or unknown. sys.argv is decoded with the file system encoding which can be wrong. Eg. see issue #4388 (ok, it's a bug, it should be fixed).

As os.environb, it would be useful to have bytes version of sys.argv to have able to decide the encoding used to decode each argument, or to manipulate bytes if we don't care about the encoding.

See also issue #8775 which propose to add a new encoding to decode sys.argv.

> sys.argv is decoded with the file system encoding

IIRC this is not exact. Py_Main signature is 
    Py_Main(int argc, wchar_t **argv)
then PyUnicode_FromWideChar is used, and there is no conversion (except from UCS4 to UCS2).
The wchar_t strings themselves are built with mbstowcs(), the file system encoding is not used.

> The wchar_t strings themselves are built with mbstowcs(), 
> the file system encoding is not used.

Oops sorry, you are right, and it's worse :-) sys.argv is decoded using the locale encoding, but subprocess & cie use the file system encoding for the reverse operation. => it doesn't work if both encodings are different (#4388, #8775).

The pseudo-code to create sys.argv on Unix is:

 # argv is a bytes list
 encoding = locale.getpreferredencoding()
 sys.argv = [arg.decode(encoding, 'surrogateescape') for arg in argv]

> As os.environb, it would be useful to have bytes version of sys.argv
> to have able to decide the encoding used to decode each argument, or
> to manipulate bytes if we don't care about the encoding.

-1. Py_Main expects wchar_t*, so no byte-oriented representation of the
command line is readily available.

"no byte-oriented representation of the command line is readily available."

Why not using the following recipe?

 encoding = locale.getpreferredencoding()
 sys.argvb = [arg.decode(encoding, 'surrogateescape') for arg in argv]

You should read .encode(), not .decode() :-/

Using that approach would work on POSIX systems.

Another problem I see is synchronizing the two. If some function strips arguments from sys.argv (because it has completed processing), sys.argvb would still keep the arguments. Of course, this could be fixed by having sys.argvb be a dynamic list (i.e. a sequence object) instead.

> Using that approach would work on POSIX systems.

As os.environb, I think that sys.argv should not exist on Windows.

> Another problem I see is synchronizing the two

os.environ and os.environb are synchronized. It would be possible to do the same with sys.argv and sys.argvb. The implement would be simplier because it's just a list, not a dict.

STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
>> Using that approach would work on POSIX systems.
> 
> As os.environb, I think that sys.argv should not exist on Windows.
> 
>> Another problem I see is synchronizing the two
> 
> os.environ and os.environb are synchronized. It would be possible to do the same with sys.argv and sys.argvb. The implement would be simplier because it's just a list, not a dict.

+1 on adding sys.argvb for systems that use char* in main().

Since r85765 (issue #4388), always use UTF-8 to decode the command line arguments on Mac OS X, not the locale encoding. Which means that the pseudo-code becomes:

 if os.name != 'nt':
     if sys.platform == 'darwin':
        encoding = 'utf-8'
     else:
        encoding = locale.getpreferredencoding()
     sys.argvb = [arg.decode(encoding, 'surrogateescape') for arg in sys.argv]

sys.argvb should be synchronized with sys.argv, as os.environb with os.environ.

Prototype (in Python) of argvb.py. Try it with: ./python -i argvb.py.

It's not possible to create sys.argvb in Python in a module loaded by Py_Initialize(), because sys.argv is created after Py_Initialize().

One year after opening the issue, I don't have any real use case. And there are technical issues to implement this feature, so I prefer just to close this issue. Reopen it if you really want it, but please give an use case ;-)