This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Interpreter returns control to cmd.exe early
Type: behavior Stage: resolved
Components: IO, Windows Versions: Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Ivan.Pozdeev, Vano, eryksun, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2016-01-24 02:29 by Ivan.Pozdeev, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (14)
msg258880 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2016-01-24 02:29
When running python.exe from windows console non-interactively, cmd.exe prompt appears immediately after starting:

C:\>python -c "import time; time.sleep(2); print 'bla-bla-bla'"

C:\>bla-bla-bla
<cursor here>

Not only this prevents from cmd to setting ERRORLEVEL to the return code, this makes it impossible to run scripts that expect input from console because Python and cmd get input lines in turns (I typed both inputs 2 times in the following example):

C:\>python -c "s=raw_input('1st:'); print s; s=raw_input('2nd:'); print s"

C:\>1st:abcdef
'abcdef' is not recognized as an internal or external command,
operable program or batch file.

C:\>abcdef
abcdef
2nd:123456
'123456' is not recognized as an internal or external command,
operable program or batch file.

C:\>123456
123456
<cursor here>
msg258881 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2016-01-24 02:53
This also happens when running interactively - so the interactive interpreter has to be run from a non-console program to be usable.

This doesn't happen with other programs (e.g. Cygwin bash) or IPython console.
msg258882 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-01-24 02:56
What version of Python are you using, where did you get it from, and what version of Windows are you using?
msg258883 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2016-01-24 03:43
Python 2.7.11 (v2.7.11:6d1b6a68f775, Dec  5 2015, 20:40:30) [MSC v.1500 64 bit (AMD64)] on win32
WinXP x64 SP2

I also tested with PowerShell 1.0. Python opens another window, but the shell's prompt also shows up immediately.
msg258884 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-01-24 04:15
That'd suggest a change in the link options since 2.7.10, probably SUBSYSTEM. Or possibly there's a config option that prevents allocating a console correctly on startup.

FWIW, I don't think WinXP 64-bit was ever considered trustworthy, so it may be a bug in the OS that never got fixed.
msg258885 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-01-24 06:43
Try forcing cmd to wait using the "start" command:

    C:\>start /b /w python -c "raise SystemExit(42)"
    C:\>echo %errorlevel%
    42
msg258904 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2016-01-25 15:59
@eryksun tried that already, same effect:

C:\>start /b /w python -c "print 'bla-bla-bl
a'; raise SystemExit(42)"

C:\>bla-bla-bla
echo %errorlevel%
0

I'll try to pinpoint the issue to an OS/OS family/update or Python version.
msg258914 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-01-25 18:08
That the wait failed to get the 42 exit code means that the "python" command (which may not actually be python.exe) is spawning a child process to run the command and not waiting for it to exit. Please try the following using the absolute path to python.exe:

    C:\>start "title" /b /w "PATH\TO\python.exe" -c "raise SystemExit(42)"
    C:\>echo %errorlevel%
    42

In this case "title" is required because cmd parses the first quoted string prior to the executable as the title. Without "title" it would parse the path to python.exe as the title and try to execute "-c".
msg258916 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2016-01-25 18:27
@eryksun That's it! "python" was actually launching a shortcut in my "shortcuts to often-used commands around the system" folder!

Thank goodness (this time, "Goodness" is Brian Curtin with 90617:a9d34685ec47, Sat May 10 12:52:59 2014 -0500 (so tell him he can add this to the list of his nicknames ;) )), the installer now has an option to add the EXEs to PATH, making this hack unneeded.
msg258922 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-01-25 21:00
By shortcut I'm guessing you mean a batch file. A regular .LNK shortcut works fine if .LNK is in PATHEXT. I sometimes use a shortcut when I need to set the default to "Run as administrator". (I add a "runas" verb to the ProgIds to add this option to the menu for .PY, .PYZ, .PYW, and .PYWZ files, but the shell doesn't allow setting the default in the advanced properties like with a shortcut.)

An alternative to adding Python to PATH is to use the py launcher and virtual environments. This is especially useful when you have multiple versions installed.
msg258933 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2016-01-25 23:26
@eryksun That is (was) an .lnk indeed, and adding ".lnk" to PATHEXT is exactly what I did to make it work. This is much more maintainable than registry hacks.

An .lnk is launched with ShellExecute which returns control immediately upon successful launch (see http://stackoverflow.com/questions/31855240/execute-exe-as-jpg/31861241#31861241 for how this happens from `cmd' prompt).
msg258945 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-01-26 04:23
> An .lnk is launched with ShellExecute which returns control
> immediately upon successful launch

cmd calls ShellExecuteEx, not ShellExecute, and it uses the flags SEE_MASK_NO_CONSOLE (0x8000, don't create a new console) and SEE_MASK_NOCLOSEPROCESS (0x0040, return a process handle if possible). 

Here's a walk-through with a debugger attached to cmd while executing a LNK shortcut to "C:\Program Files\Python27\python.exe": 

    C:\Temp>.\python.lnk -c "import sys,time;time.sleep(60);sys.exit(42)"
    Breakpoint 0 hit
    SHELL32!ShellExecuteExW:
    00007ff9`a5710e20 48895c2408      mov     qword ptr [rsp+8],rbx
                                        ss:000000a3`eb3af3a0=000000a3eb597ca0
    0:000> ; as /x info @rcx
    0:000> ; as /x sz @@(*((unsigned long *)${info}))
    0:000> bd 2,3,4; pt; be 2,3,4

ShellExecuteEx returns the process handle:

    0:000> ?? *((void **)(${info} + ${sz} - 8)); * hProcess
    void * 0x00000000`000002fc
    0:000> !handle 2fc
    Handle 2fc
      Type          Process
    0:000> g

cmd uses the handle to read the ImageSubsystem type from the process environment block (PEB), for which 3 is a console process and 2 is a GUI process.

    Breakpoint 1 hit
    cmd!GetProcessSubsystemType:
    00007ff7`a133faf4 48895c2410      mov     qword ptr [rsp+10h],rbx
                                        ss:000000a3`eb3af458=000000a3eb585640
    0:000> g
    Breakpoint 2 hit
    KERNELBASE!ReadProcessMemory:
    00007ff9`a49ac230 4883ec48        sub     rsp,48h
    0:000> as /x buf @r8
    0:000> pt
    KERNELBASE!ReadProcessMemory+0x2b:
    00007ff9`a49ac25b c3              ret

    0:000> ?? ((ntdll!_PEB *)${buf})->ImageSubsystem
    unsigned long 3
    0:000> g

Since it's a console process, cmd waits and queries the exit code.

    Breakpoint 3 hit    
    KERNELBASE!WaitForSingleObject:
    00007ff9`a49840c0 4533c0          xor     r8d,r8d
    0:000> r rcx
    rcx=00000000000002fc
    0:000> g

    Breakpoint 4 hit
    KERNELBASE!GetExitCodeProcess:
    00007ff9`a49c46d0 4053            push    rbx
    0:000> as /x rc @rdx
    0:000> pt
    KERNELBASE!GetExitCodeProcess+0x3a:
    00007ff9`a49c470a c3              ret
    0:000> ?? *((unsigned long *)${rc})
    unsigned long 0x2a
    0:000> ? 0x2a
    Evaluate expression: 42 = 00000000`0000002a

It sets the exit code in the 'hidden' environment variable "=ExitCode" as a unsigned hexadecimal number.

    C:\Temp>echo %=ExitCode%
    0000002A

You can also query the signed value using the pseudo environment variable "errorlevel".

    C:\Temp>echo %errorlevel%
    42
msg259181 - (view) Author: Ivan Pozdeev (Vano) Date: 2016-01-29 00:42
Here, ShellExecuteExW is indeed called, with the specified parameters, 
but the hProcess member is still NULL after the call. This must be the 
reason for the discrepancy.
msg259192 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-01-29 03:56
Ensure that the .lnk target is the expected path to python.exe:

    C:\Temp>py -3 -c ^
    More? "from win32com import client^
    More?
    More? lnk = client.Dispatch('WScript.Shell').^
    More? CreateShortCut('python.lnk')^
    More?
    More? print('target:\n%s' % lnk.Targetpath)
    target:
    C:\Program Files\Python27\python.exe

You can attach a debugger to cmd and set a breakpoint on CreateProcessW. The first time it's called is for the failed attempt to execute the .lnk as a PE image (cmd doesn't make assumptions about file extensions). The 2nd time, if called, should be from a ShellExecute worker thread. For example (this time using Windows 7):

    C:\Temp>.\python.lnk -c "import sys;sys.exit(42)"

    Breakpoint 0 hit
    kernel32!CreateProcessW:
    00000000`76c307a0 4883ec68        sub     rsp,68h
    0:000> du @rdx
    00000000`003942f0  ".\python.lnk  -c "import sys;sys"
    00000000`00394330  ".exit(42)""
    0:000> g

    Breakpoint 0 hit
    kernel32!CreateProcessW:
    00000000`76c307a0 4883ec68        sub     rsp,68h
    0:001> du @rdx
    00000000`003d3640  ""C:\Program Files\Python27\pytho"
    00000000`003d3680  "n.exe"  -c "import sys;sys.exit("
    00000000`003d36c0  "42)""
    0:001> k2
    Child-SP          RetAddr           Call Site
    00000000`0149e4e8 000007fe`fde3b9c3 kernel32!CreateProcessW
    00000000`0149e4f0 000007fe`fde3b72b SHELL32!_SHCreateProcess+0x32b
    0:001> g

    C:\Temp>echo %errorlevel%
    42
History
Date User Action Args
2022-04-11 14:58:26adminsetgithub: 70377
2016-01-29 03:56:43eryksunsetmessages: + msg259192
2016-01-29 00:42:37Vanosetnosy: + Vano
messages: + msg259181
2016-01-26 04:23:21eryksunsetmessages: + msg258945
2016-01-25 23:26:51Ivan.Pozdeevsetmessages: + msg258933
2016-01-25 21:00:53eryksunsetmessages: + msg258922
stage: resolved
2016-01-25 18:27:22Ivan.Pozdeevsetstatus: open -> closed
resolution: not a bug
messages: + msg258916
2016-01-25 18:08:28eryksunsetmessages: + msg258914
2016-01-25 15:59:41Ivan.Pozdeevsetmessages: + msg258904
2016-01-24 06:43:02eryksunsetnosy: + eryksun
messages: + msg258885
2016-01-24 04:15:23steve.dowersetmessages: + msg258884
2016-01-24 03:43:27Ivan.Pozdeevsetmessages: + msg258883
2016-01-24 02:56:24zach.waresetnosy: + tim.golden, steve.dower, zach.ware, paul.moore
messages: + msg258882
components: + Windows
2016-01-24 02:53:57Ivan.Pozdeevsetmessages: + msg258881
title: Non-interactive interpreter returns control to cmd.exe early -> Interpreter returns control to cmd.exe early
2016-01-24 02:29:02Ivan.Pozdeevcreate