Title: Support bytes for os.exec*()
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.1
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, loewis, pitrou, vstinner
Priority: high Keywords: patch

Created on 2008-10-03 23:38 by vstinner, last changed 2010-03-03 22:09 by vstinner. This issue is now closed.

File name Uploaded Description Edit
os_exec_bytes-2.patch vstinner, 2008-10-08 00:38
Messages (11)
msg74283 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-10-03 23:38
os.exec*() functions doesn't support bytes if the program name doesn't 
use absolute path. The problem is that PATH is used to complete the 
full path but Python3 disallows bytes+str (which is a good thing!). 

python -c "import os; os.execvp('pwd', 'pwd')"
Traceback (most recent call last):
  File "/home/haypo/prog/py3k/Lib/", line 328, in execvp
    _execvpe(file, args)
  File "/home/haypo/prog/py3k/Lib/", line 364, in _execvpe
    func(fullname, *argrest)
TypeError: execv() arg 2 must be a tuple or list

Attached patch allows bytes in os.exec*(). It converts each directory 
of PATH using sys.getfilesystemencoding().
msg74304 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-10-04 08:03
This is incorrect. On Windows, if the path contains characters with no
representation in CP_ACP, encoding the path as bytes makes it invalid.
msg74313 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-10-04 11:31
The fix can be changed to be specific to POSIX system:
+    if name == 'posix' \
+    and isinstance(file, bytes):
+        encoding = sys.getfilesystemencoding()
+        PATH = (bytes(dir, encoding) for dir in PATH)

My example was incorrect. The good example is:
   python -c "import os; os.execvp('pwd', ['pwd'])
msg74449 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-10-07 13:43
Can you come up with a test case?

Also, it would probably be good to document what parameters can have
what datatypes in the exec family of functions.
msg74451 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-10-07 14:19
The patch is incomplete. It allows bytes for the arguments but not for 
the environment variables: posix_execve() in Modules/posixmodule.c 
 PyArg_Parse(key "s", &k)
 PyArg_Parse(val "s", &v)
msg74503 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-10-08 00:38
Improved patch:
 - support bytes in the env dictionary using the file system default 
 - support bytes for program arguments but only on a POSIX system

Document is not updated yet.
msg78681 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-01-01 02:19
Hmm, I think the supported types should be the same for all platforms,
otherwise it creates unnecessary headaches.

Perhaps, in addition to the proposed behaviour on Posix (convert
everything to bytes if the program name is a bytes object), the reverse
could be done under Windows (convert the program name to str if it is a
bytes object).
msg78718 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-01-01 18:56
> Hmm, I think the supported types should be the same for all
> platforms, otherwise it creates unnecessary headaches.

I'm don't know Windows very well, but I read many times that Windows 
uses unicode everywhere. I'm unable to decide if it's a good thing or 
not to support also bytes on Windows, and anyway I'm unable to write 
such patch.

See also issue #4036 (bytes for subprocess.Popen), especially Martin's 
message msg74305 (Martin is our Windows expert ;-)).
msg78739 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2009-01-01 23:44
Every function of the Windows API comes in pair: an Ansi function (which 
accepts char* names) and a Wide function (which accepts wchar_t* names; 
Py_UNICODE* can be passed as-is)

Don't perform conversion on Windows, just call the proper function.
msg78829 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2009-01-02 16:15
Any discussion in the tracker should be deferred until a PEP has been
written, discussed, and accepted. Then the question whether to accept a
specific patch shouldn't be a matter of taste, but should follow from
the general rules that the PEP has set up.
msg100355 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-03-03 22:09
Fixed by r72313 (PEP 383).
Date User Action Args
2010-03-03 22:09:01vstinnersetstatus: open -> closed
keywords: patch, patch
resolution: fixed
messages: + msg100355
2009-05-17 02:43:02ajaksu2setversions: + Python 3.1, - Python 3.0
nosy: loewis, amaury.forgeotdarc, pitrou, vstinner
priority: high
components: + Library (Lib)
keywords: patch, patch
type: enhancement
2009-01-02 16:16:00loewissetkeywords: patch, patch
messages: + msg78829
2009-01-01 23:44:20amaury.forgeotdarcsetkeywords: patch, patch
nosy: + amaury.forgeotdarc
messages: + msg78739
2009-01-01 18:56:02vstinnersetkeywords: patch, patch
messages: + msg78718
2009-01-01 02:19:37pitrousetkeywords: patch, patch
nosy: + pitrou
messages: + msg78681
2008-10-08 00:43:39vstinnerlinkissue4036 dependencies
2008-10-08 00:39:37vstinnersetfiles: - os_exec_bytes.patch
2008-10-08 00:38:29vstinnersetkeywords: patch, patch
files: + os_exec_bytes-2.patch
messages: + msg74503
2008-10-07 14:19:54vstinnersetkeywords: patch, patch
messages: + msg74451
2008-10-07 13:43:35loewissetkeywords: patch, patch
messages: + msg74449
2008-10-04 11:31:27vstinnersetkeywords: patch, patch
messages: + msg74313
2008-10-04 08:03:08loewissetkeywords: patch, patch
nosy: + loewis
messages: + msg74304
2008-10-03 23:38:23vstinnercreate