classification
Title: python hung or stuck somtimes randomly on windows server 2008R2
Type: crash Stage: resolved
Components: Windows Versions: Python 3.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Cao Hongfu, eryksun, izbyshev, paul.moore, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords:

Created on 2018-12-05 12:51 by Cao Hongfu, last changed 2018-12-06 02:53 by eryksun. This issue is now closed.

Messages (7)
msg331105 - (view) Author: Cao Hongfu (Cao Hongfu) Date: 2018-12-05 12:51
Recently, python frequently(but randomly) hung or stuck at initialization(when I click the python.exe or use python cmd prompt) on my server(running windows server 2008R2), But everything is fine on my windows 7 PC). I tried reinstall python, but not working(also tried 3.6).

I tried process-explorer and found that the normal python process allocated about 12MB memory but the stuck one only allocated 8MB memory.

here is the stack information for stuck python process(having 3 threads):
------------------thread1------------------
ntdll.dll!ZwWaitForSingleObject+0xa
ntdll.dll!RtlImageDirectoryEntryToData+0x118
ntdll.dll!RtlEnterCriticalSection+0xd1
ntdll.dll!EtwDeliverDataBlock+0x777
ntdll.dll!LdrLoadDll+0xed
!TlsGetValue+0x4756
!UuidCreate+0x1b00
!I_RpcBindingIsServerLocal+0x12899
!RegEnumKeyExW+0x13a
!RegEnumKeyExW+0xbe
!RpcBindingFree+0x320
!RpcAsyncRegisterInfo+0x10ff
!Ndr64AsyncClientCall+0x9da
!Ndr64AsyncClientCall+0xc9b
!NdrClientCall3+0xf5
!LsaOpenPolicy+0xb9
!LsaOpenPolicy+0x56
!LookupPrivilegeValueW+0x6f
!LookupPrivilegeValueA+0x84
!PyNamespace_New+0xd4
!PyCodec_LookupTextEncoding+0xb5
!PyObject_SetAttrId+0x21e
!PyMethodDef_RawFastCallDict+0x115
!PyObject_SetAttr+0x352
!PyEval_EvalFrameDefault+0x1182
!PyEval_EvalCodeWithName+0x1a0
!PyMethodDef_RawFastCallKeywords+0xc32
!PyEval_EvalFrameDefault+0x4b1
!PyMethodDef_RawFastCallKeywords+0xa77
!PyEval_EvalFrameDefault+0x913
!PyMethodDef_RawFastCallKeywords+0xa77
!PyEval_EvalFrameDefault+0x4b1
!PyMethodDef_RawFastCallKeywords+0xa77
!PyEval_EvalFrameDefault+0x4b1
!PyMethodDef_RawFastCallKeywords+0xa77
!PyEval_EvalFrameDefault+0x913
!PyMethodDef_RawFastCallKeywords+0xa77
!PyEval_EvalFrameDefault+0x4b1
!PyMethodDef_RawFastCallKeywords+0xa77
!PyEval_EvalFrameDefault+0x913
!PyFunction_FastCallDict+0xdd
!PyObject_CallMethod+0xef
!PyObject_CallMethod+0xa2
!PyObject_CallMethod+0x3c
!PyTime_MulDiv+0x47
!Py_InitializeMainInterpreter+0x95
!PyMainInterpreterConfig_Read+0x309
!PyMapping_SetItemString+0x306
!PyBytes_AsString+0x142
!Py_Main+0x52
!BaseThreadInitThunk+0xd
ntdll.dll!RtlUserThreadStart+0x1d
------------------thread2------------------
ntdll.dll!ZwWaitForSingleObject+0xa
ntdll.dll!RtlImageDirectoryEntryToData+0x118
ntdll.dll!RtlEnterCriticalSection+0xd1
!UuidCreate+0x1ae2
!NdrFullPointerQueryPointer+0x35d
!LsaLookupGetDomainInfo+0xb8
!RpcBindingFree+0x320
!RpcAsyncRegisterInfo+0x10ff
!Ndr64AsyncClientCall+0x9da
!Ndr64AsyncClientCall+0xc9b
!NdrClientCall3+0xf5
!LsaLookupOpenLocalPolicy+0x41
!LookupAccountNameLocalW+0xaf
!LookupAccountSidLocalW+0x25
!LookupAccountSidW+0x57
!MBCGlobal::get_proc_user_name+0x1f7
!MBCGlobal::init+0x240a
!HDirSnap::operator=+0xda
!LVPVTBase::to_file+0x46ef
ntdll.dll!RtlDeactivateActivationContextUnsafeFast+0x34e
ntdll.dll!EtwDeliverDataBlock+0xa44
ntdll.dll!LdrLoadDll+0xed
!TlsGetValue+0x4756
!PublicService+0x13ec
!BaseThreadInitThunk+0xd
ntdll.dll!RtlUserThreadStart+0x1d
------------------thread3------------------
ntdll.dll!ZwWaitForSingleObject+0xa
ntdll.dll!RtlImageDirectoryEntryToData+0x118
ntdll.dll!RtlEnterCriticalSection+0xd1
ntdll.dll!LdrQueryModuleServiceTags+0x13f
ntdll.dll!CsrIdentifyAlertableThread+0x9d
ntdll.dll!EtwSendNotification+0x16d
ntdll.dll!RtlQueryProcessDebugInformation+0x371
ntdll.dll!EtwDeliverDataBlock+0xf00
!BaseThreadInitThunk+0xd
ntdll.dll!RtlUserThreadStart+0x1d


here is the stack info for normal python process(have 2 threads)
------------------thread1------------------
ntdll.dll!ZwRequestWaitReplyPort+0xa
kernel32.dll!GetConsoleMode+0xf8
kernel32.dll!VerifyConsoleIoHandle+0x281
kernel32.dll!ReadConsoleW+0xbc
python37.dll!PyOS_Readline+0x4f4
python37.dll!PyOS_Readline+0x333
python37.dll!PyOS_Readline+0xfa
python37.dll!PyErr_NoMemory+0xc228
python37.dll!PyUnicode_AsUnicode+0x553
python37.dll!PyUnicode_AsUnicode+0x9c
python37.dll!PyParser_ParseFileObject+0x86
python37.dll!PyParser_ASTFromFileObject+0x82
python37.dll!PyRun_InteractiveOneObject+0x24a
python37.dll!PyRun_InteractiveLoopFlags+0xf6
python37.dll!PyRun_AnyFileExFlags+0x45
python37.dll!Py_UnixMain+0x50b
python37.dll!Py_UnixMain+0x5b3
python37.dll!PyErr_NoMemory+0x195a4
python37.dll!PyBytes_AsString+0x14f
python37.dll!Py_Main+0x52
python.exe+0x1258
kernel32.dll!BaseThreadInitThunk+0xd
ntdll.dll!RtlUserThreadStart+0x1d
------------------thread2------------------
ntdll.dll!NtWaitForMultipleObjects+0xa
ntdll.dll!RtlIsCriticalSectionLockedByThread+0xd4d
kernel32.dll!BaseThreadInitThunk+0xd
ntdll.dll!RtlUserThreadStart+0x1d


One of my friend say that this may be issues with
https://support.microsoft.com/en-us/help/2545627/a-multithreaded-application-might-crash-in-windows-7-or-in-windows-ser

Thx.
msg331132 - (view) Author: Alexey Izbyshev (izbyshev) * (Python triager) Date: 2018-12-05 15:32
You might try to check the list of DLLs loaded into the stuck python process and find third-party ones (e.g., antivirus). If there are any, disable the third-party software and try again.
msg331168 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-05 21:15
You can try to use faulthandler.dump_traceback_later() with a file to get the traceback of the stuck threads.

------------------thread1------------------
...
!UuidCreate+0x1b00
...
!RegEnumKeyExW+0xbe
...
!LookupPrivilegeValueA+0x84
!PyNamespace_New+0xd4
!PyCodec_LookupTextEncoding+0xb5
!PyObject_SetAttrId+0x21e
...
!PyObject_CallMethod+0x3c
!PyTime_MulDiv+0x47
!Py_InitializeMainInterpreter+0x95
!PyMainInterpreterConfig_Read+0x309
!PyMapping_SetItemString+0x306
!PyBytes_AsString+0x142
!Py_Main+0x52
!BaseThreadInitThunk+0xd

This traceback doesn't make sense:

* Py_Main() doesn't call directly PyBytes_AsString().
* PyTime_MulDiv shouldn't call PyObject_CallMethod().
* I don't see how PyNamespace_New() can call LookupPrivilegeValueA()
msg331175 - (view) Author: Alexey Izbyshev (izbyshev) * (Python triager) Date: 2018-12-05 21:35
How is it possible to use faulthandler if the interpreter hasn't even started yet?
msg331195 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2018-12-05 23:56
> This traceback doesn't make sense

Enable the installer options for the debug binaries and symbol files. Ensure that this installs the *_d.[exe|dll] debug binaries and also the *.pdb symbol files for the release and debug builds. 

Run the debug build via python_d.exe to check whether it aborts from a failed assertion. If it still hangs, in Process Explorer open the dialog for Options -> Configure Symbols and add the installation directory of Python 3.7 to the beginning of the "Symbols path". At least now your stack traces should have proper symbols instead of only the names of exported functions, for what that's worth. It's a poor substitute for a [mini]dump file and a debugger.
msg331202 - (view) Author: Cao Hongfu (Cao Hongfu) Date: 2018-12-06 01:45
I tried windows resource manager and found that stuck python process does not have load these three DLL files(or stuck on loading these DLL files):
MozartBreathBolo.dll
MozartBreathNet.dll
MozartBreathProcess.dll.
These DLL files were created by our company's supervisory software,I contact the IT guy and disabled it. Now everything is fine.
Without the supervisory software, the normal python should allocate only 8MB memeory.
THX you guys, and sorry for wasting your time~
msg331203 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2018-12-06 02:53
> I don't see how PyNamespace_New() can call LookupPrivilegeValueA()

For the record, in the 3.7.1 release build, `PyNamespace_New + d4` is in enable_symlink (Modules/posixmodule.c), which gets called when the nt (aka posix) module gets initialized. It's the return address for the LookupPrivilegeValueA call:

    0:000> u (python37!PyNamespace_New + d4 - 6) l2
    python37!enable_symlink+0x42:
    00007ffd`e4b9a356 
        ff153ccd1500    call    qword ptr 
        [python37!_imp_LookupPrivilegeValueA (00007ffd`e4cf7098)]
    00007ffd`e4b9a35c 
        85c0            test    eax,eax

It's just that Cao didn't have the python37.pdb symbol file available in Process Explorer.
History
Date User Action Args
2018-12-06 02:53:24eryksunsetmessages: + msg331203
2018-12-06 01:46:49Cao Hongfusetstatus: open -> closed
stage: resolved
2018-12-06 01:45:20Cao Hongfusetresolution: not a bug
messages: + msg331202
2018-12-05 23:56:04eryksunsetnosy: + eryksun
messages: + msg331195
2018-12-05 21:35:59izbyshevsetmessages: + msg331175
2018-12-05 21:15:21vstinnersetnosy: + vstinner
messages: + msg331168
2018-12-05 15:32:59izbyshevsetnosy: + izbyshev
messages: + msg331132
2018-12-05 12:51:48Cao Hongfucreate