msg245700 - (view) |
Author: Florian Bruhin (The Compiler) * |
Date: 2015-06-23 20:37 |
Trying to call a 32bit Python via subprocess from a 64bit one works as expected:
Python 3.4.2 (v3.4.2:ab2c023a9432, Oct 6 2014, 22:16:31) [MSC v.1600 64 bit (AMD64)] on win32
[...]
>>> subprocess.check_call([r'C:\Python34_x32\python.exe', '-c', 'print("Hello World")'])
Hello World
However, when passing the `env` parameter, even just as `env=os.environ`, things break:
>>> subprocess.check_call([r'C:\Python34_x32\python.exe', '-c', 'print("Hello World")'], env=os.environ)
Fatal Python error: Failed to initialize Windows random API (CryptoGen)
Starting a 64bit Python from the 64bit one works fine:
>>> subprocess.check_call([r'C:\Python34\python.exe', '-c', 'print("Hello World")'], env=os.environ)
Hello World
According to issue20614 the "Fatal Python error" happens when SYSTEMROOT is not set, but that's definitely set in the original environment.
Looking at the code, it just passes env to Windows' CreateProcess[1] - I guess this does *something* different in this scenario when None/NULL is given as compared to the os.environ dict?
This breaks virtualenv with --python for me:
C:\Users\florian\tmp>C:\Python34\python.exe -m virtualenv --python=C:\Python34_x32\python.exe venv3
Running virtualenv with interpreter C:\Python34_x32\python.exe
Fatal Python error: Failed to initialize Windows random API (CryptoGen)
[1] https://msdn.microsoft.com/en-us/library/windows/desktop/ms682425(v=vs.85).aspx
|
msg245703 - (view) |
Author: Eryk Sun (eryksun) * |
Date: 2015-06-24 02:08 |
I can't reproduce this in Windows 7 or 10 using Python 3.4. What gets printed for the following?
import os
import subprocess
cmd32 = os.path.join(os.environ['SYSTEMROOT'], 'SysWOW64', 'cmd.exe')
subprocess.call('{} /c set SYSTEMROOT'.format(cmd32), env=os.environ)
|
msg245706 - (view) |
Author: Florian Bruhin (The Compiler) * |
Date: 2015-06-24 03:27 |
That gives me "Environment variable SYSTEMROOT not defined" which would explain the "Fatal Python error" I'm seeing.
|
msg245717 - (view) |
Author: Florian Bruhin (The Compiler) * |
Date: 2015-06-24 04:14 |
Sorry I missed this - I can reproduce this on Windows 8.1, but not on Windows 7. I hope I'll be able to try another Windows 8.1 machine today.
SYSTEMROOT is definitely set in the original environment:
>>> os.environ['SYSTEMROOT']
'C:\\Windows'
>>> subprocess.call('{} /c set SYSTEMROOT'.format(cmd32), env=os.environ)
Environment variable SYSTEMROOT not defined
1
>>> subprocess.call('{} /c set SYSTEMROOT'.format(cmd32))
SystemRoot=C:\Windows
0
It seems only a minimal set of environment variables are set in the spawned process:
>>> subprocess.call('{} /c set'.format(cmd32), env=os.environ)
COMSPEC=C:\Windows\SysWOW64\cmd.exe
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.JS;.WS;.MSC
PROMPT=$P$G
|
msg245722 - (view) |
Author: Eryk Sun (eryksun) * |
Date: 2015-06-24 05:51 |
> It seems only a minimal set of environment variables are set
Apparently the initial environment is empty. The values you see are defaults set by cmd.exe when it starts. It also sets the 'hidden' variable "=C:" to the current directory on the C: drive, which you can see via set "".
As a workaround, try running the command using shell=True (i.e. cmd /c). This should let you modify the environment passed to 64-bit cmd.exe, which will be inherited by the grandchild process that it creates.
|
msg245724 - (view) |
Author: Florian Bruhin (The Compiler) * |
Date: 2015-06-24 07:09 |
I've now updated to Python 3.4.3 and it's broken there as well.
I tried on two other Windows 8.1 machines (with Python 3.4.3 and 3.4.1 respectively), and I can't reproduce there either...
It works with shell=True indeed.
I wonder why this is only broken on that one machine... It's a virtual machine which only has Firefox, Python, PyQt and a few Python packages installed - otherwise it's vanilla.
|
msg250829 - (view) |
Author: Florian Bruhin (The Compiler) * |
Date: 2015-09-16 08:00 |
I just ran into this again - when trying to run `git` via subprocess with "env" set, I got:
# Start the process
try:
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
# no special security
None, None,
int(not close_fds),
creationflags,
env,
cwd,
> startupinfo)
E FileNotFoundError: [WinError 2] The system cannot find the file specified
This only seems to happen when starting my Python process in cmd.exe, not when it's started via my buildbot (CI).
Again, when passing shell=True everything worked - except when passing `cwd` as well, then it's broken again.
|
msg250843 - (view) |
Author: R. David Murray (r.david.murray) * |
Date: 2015-09-16 12:29 |
Sounds like there's something in the cwd that is making the difference (this being Windows which looks for things in the cwd always).
|
msg250844 - (view) |
Author: Florian Bruhin (The Compiler) * |
Date: 2015-09-16 12:33 |
> Sounds like there's something in the cwd that is making the difference (this being Windows which looks for things in the cwd always).
The cwd is an empty temporary directory. And that still wouldn't explain why passing env=os.environ breaks it as well, and why it only breaks in cmd.exe, and only when launching a 32bit process ;)
|
msg250848 - (view) |
Author: Eryk Sun (eryksun) * |
Date: 2015-09-16 14:14 |
The issue as I understand it is that, on this particular Windows 8.1 system, passing a non-NULL lpEnvironment to CreateProcess works when starting a native 64-bit executable, but fails when starting a 32-bit executable via the WOW64 system. The child process instead gets an empty environment block.
As an additional check, run the following command in 64-bit cmd.exe:
start "" /I "%SystemRoot%\SysWOW64\cmd.exe"
The /I option of the start command passes the shell's original environment to CreateProcess, so it should exhibit the same empty-environment problem when starting 32-bit cmd.exe. In this case you'll get cmd's default environment, which includes COMSPEC, PATHEXT, and PROMPT.
Since inheriting the current environment works in all cases and passing a custom environment works for 64-bit executables, the workaround that I suggested is to use shell=True to pass your custom environment to the shell. The 32-bit executable thus inherits the custom environment from the shell. If using shell=True is a security concern, then you can replace it with a Python script that executes and waits for the child process.
> when passing shell=True everything worked - except
> when passing `cwd` as well, then it's broken again.
Since the shell executes the file, a relative path is resolved against the shell's working directory. If you set the latter via the cwd parameter, then pass the file's path as either relative to the cwd path or as a fully qualified path.
|
|
Date |
User |
Action |
Args |
2022-04-11 14:58:18 | admin | set | github: 68681 |
2021-02-24 18:37:41 | eryksun | set | status: open -> closed resolution: third party stage: resolved |
2015-09-16 14:14:18 | eryksun | set | messages:
+ msg250848 |
2015-09-16 12:33:34 | The Compiler | set | messages:
+ msg250844 |
2015-09-16 12:29:44 | r.david.murray | set | nosy:
+ r.david.murray messages:
+ msg250843
|
2015-09-16 08:02:11 | vstinner | set | nosy:
+ vstinner
|
2015-09-16 08:00:21 | The Compiler | set | messages:
+ msg250829 |
2015-06-24 07:09:45 | The Compiler | set | messages:
+ msg245724 |
2015-06-24 05:51:58 | eryksun | set | messages:
+ msg245722 |
2015-06-24 04:14:47 | The Compiler | set | messages:
+ msg245717 title: subprocess with env=os.environ fails with "fatal python error" when calling 32-bit python from 64-bit one on Windows -> subprocess with env=os.environ doesn't preserve environment variables when calling a 32bit process on Windows 8.1 |
2015-06-24 03:27:11 | The Compiler | set | messages:
+ msg245706 |
2015-06-24 02:08:46 | eryksun | set | nosy:
+ eryksun messages:
+ msg245703
|
2015-06-23 20:37:07 | The Compiler | create | |