classification
Title: subprocess with env=os.environ doesn't preserve environment variables when calling a 32bit process on Windows 8.1
Type: Stage:
Components: Library (Lib), Windows Versions: Python 3.4
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: The Compiler, astrand, eryksun, paul.moore, r.david.murray, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords:

Created on 2015-06-23 20:37 by The Compiler, last changed 2015-09-16 14:14 by eryksun.

Messages (10)
msg245700 - (view) Author: Florian Bruhin (The Compiler) Date: 2015-06-23 20:37
Trying to call a 32bit Python via subprocess from a 64bit one works as expected:

    Python 3.4.2 (v3.4.2:ab2c023a9432, Oct  6 2014, 22:16:31) [MSC v.1600 64 bit (AMD64)] on win32
    [...]
    >>> subprocess.check_call([r'C:\Python34_x32\python.exe', '-c', 'print("Hello World")'])
    Hello World

However, when passing the `env` parameter, even just as `env=os.environ`, things break:

    >>> subprocess.check_call([r'C:\Python34_x32\python.exe', '-c', 'print("Hello World")'], env=os.environ)
    Fatal Python error: Failed to initialize Windows random API (CryptoGen)

Starting a 64bit Python from the 64bit one works fine:

    >>> subprocess.check_call([r'C:\Python34\python.exe', '-c', 'print("Hello World")'], env=os.environ)
    Hello World

According to issue20614 the "Fatal Python error" happens when SYSTEMROOT is not set, but that's definitely set in the original environment.

Looking at the code, it just passes env to Windows' CreateProcess[1] - I guess this does *something* different in this scenario when None/NULL is given as compared to the os.environ dict?

This breaks virtualenv with --python for me:

    C:\Users\florian\tmp>C:\Python34\python.exe -m virtualenv --python=C:\Python34_x32\python.exe venv3
    Running virtualenv with interpreter C:\Python34_x32\python.exe
    Fatal Python error: Failed to initialize Windows random API (CryptoGen)

[1] https://msdn.microsoft.com/en-us/library/windows/desktop/ms682425(v=vs.85).aspx
msg245703 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2015-06-24 02:08
I can't reproduce this in Windows 7 or 10 using Python 3.4. What gets printed for the following?

import os
import subprocess

cmd32 = os.path.join(os.environ['SYSTEMROOT'], 'SysWOW64', 'cmd.exe')
subprocess.call('{} /c set SYSTEMROOT'.format(cmd32), env=os.environ)
msg245706 - (view) Author: Florian Bruhin (The Compiler) Date: 2015-06-24 03:27
That gives me "Environment variable SYSTEMROOT not defined" which would explain the "Fatal Python error" I'm seeing.
msg245717 - (view) Author: Florian Bruhin (The Compiler) Date: 2015-06-24 04:14
Sorry I missed this - I can reproduce this on Windows 8.1, but not on Windows 7. I hope I'll be able to try another Windows 8.1 machine today.

SYSTEMROOT is definitely set in the original environment:

    >>> os.environ['SYSTEMROOT']
    'C:\\Windows'
    >>> subprocess.call('{} /c set SYSTEMROOT'.format(cmd32), env=os.environ)
    Environment variable SYSTEMROOT not defined
    1
    >>> subprocess.call('{} /c set SYSTEMROOT'.format(cmd32))
    SystemRoot=C:\Windows
    0

It seems only a minimal set of environment variables are set in the spawned process:

    >>> subprocess.call('{} /c set'.format(cmd32), env=os.environ)
    COMSPEC=C:\Windows\SysWOW64\cmd.exe
    PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.JS;.WS;.MSC
    PROMPT=$P$G
msg245722 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2015-06-24 05:51
> It seems only a minimal set of environment variables are set 

Apparently the initial environment is empty. The values you see are defaults set by cmd.exe when it starts. It also sets the 'hidden' variable "=C:" to the current directory on the C: drive, which you can see via set "". 

As a workaround, try running the command using shell=True (i.e. cmd /c). This should let you modify the environment passed to 64-bit cmd.exe, which will be inherited by the grandchild process that it creates.
msg245724 - (view) Author: Florian Bruhin (The Compiler) Date: 2015-06-24 07:09
I've now updated to Python 3.4.3 and it's broken there as well.

I tried on two other Windows 8.1 machines (with Python 3.4.3 and 3.4.1 respectively), and I can't reproduce there either...

It works with shell=True indeed.

I wonder why this is only broken on that one machine... It's a virtual machine which only has Firefox, Python, PyQt and a few Python packages installed - otherwise it's vanilla.
msg250829 - (view) Author: Florian Bruhin (The Compiler) Date: 2015-09-16 08:00
I just ran into this again - when trying to run `git` via subprocess with "env" set, I got:

        # Start the process
        try:
            hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                                     # no special security
                                     None, None,
                                     int(not close_fds),
                                     creationflags,
                                     env,
                                     cwd,
>                                    startupinfo)
E                                    FileNotFoundError: [WinError 2] The system cannot find the file specified

This only seems to happen when starting my Python process in cmd.exe, not when it's started via my buildbot (CI).

Again, when passing shell=True everything worked - except when passing `cwd` as well, then it's broken again.
msg250843 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-09-16 12:29
Sounds like there's something in the cwd that is making the difference (this being Windows which looks for things in the cwd always).
msg250844 - (view) Author: Florian Bruhin (The Compiler) Date: 2015-09-16 12:33
> Sounds like there's something in the cwd that is making the difference (this being Windows which looks for things in the cwd always).

The cwd is an empty temporary directory. And that still wouldn't explain why passing env=os.environ breaks it as well, and why it only breaks in cmd.exe, and only when launching a 32bit process ;)
msg250848 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2015-09-16 14:14
The issue as I understand it is that, on this particular Windows 8.1 system, passing a non-NULL lpEnvironment to CreateProcess works when starting a native 64-bit executable, but fails when starting a 32-bit executable via the WOW64 system. The child process instead gets an empty environment block. 

As an additional check, run the following command in 64-bit cmd.exe:

    start "" /I "%SystemRoot%\SysWOW64\cmd.exe"

The /I option of the start command passes the shell's original environment to CreateProcess, so it should exhibit the same empty-environment problem when starting 32-bit cmd.exe. In this case you'll get cmd's default environment, which includes COMSPEC, PATHEXT, and PROMPT.

Since inheriting the current environment works in all cases and passing a custom environment works for 64-bit executables, the workaround that I suggested is to use shell=True to pass your custom environment to the shell. The 32-bit executable thus inherits the custom environment from the shell. If using shell=True is a security concern, then you can replace it with a Python script that executes and waits for the child process. 

> when passing shell=True everything worked - except 
> when passing `cwd` as well, then it's broken again.

Since the shell executes the file, a relative path is resolved against the shell's working directory. If you set the latter via the cwd parameter, then pass the file's path as either relative to the cwd path or as a fully qualified path.
History
Date User Action Args
2015-09-16 14:14:18eryksunsetmessages: + msg250848
2015-09-16 12:33:34The Compilersetmessages: + msg250844
2015-09-16 12:29:44r.david.murraysetnosy: + r.david.murray
messages: + msg250843
2015-09-16 08:02:11vstinnersetnosy: + vstinner
2015-09-16 08:00:21The Compilersetmessages: + msg250829
2015-06-24 07:09:45The Compilersetmessages: + msg245724
2015-06-24 05:51:58eryksunsetmessages: + msg245722
2015-06-24 04:14:47The Compilersetmessages: + msg245717
title: subprocess with env=os.environ fails with "fatal python error" when calling 32-bit python from 64-bit one on Windows -> subprocess with env=os.environ doesn't preserve environment variables when calling a 32bit process on Windows 8.1
2015-06-24 03:27:11The Compilersetmessages: + msg245706
2015-06-24 02:08:46eryksunsetnosy: + eryksun
messages: + msg245703
2015-06-23 20:37:07The Compilercreate