classification
Title: can't make IDLEX work with python._pth and python-3.6.0b2
Type: crash Stage: resolved
Components: Windows Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: Big Stone, eryksun, illagrenan, paul.moore, python-dev, roger.serwy, steve.dower, terry.reedy, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2016-10-24 20:45 by Big Stone, last changed 2017-03-31 16:36 by dstufft. This issue is now closed.

Files
File name Uploaded Description Edit
Report.wer Big Stone, 2016-11-01 11:22
Pull Requests
URL Status Linked Edit
PR 552 closed dstufft, 2017-03-31 16:36
Messages (29)
msg279338 - (view) Author: Big Stone (Big Stone) Date: 2016-10-24 20:45
on WinPython-64bit-3.6.0.0Zerorc2.exe, python-3.6.0b2 based, I can't get IDLEX working with "python._pth".

If I put "Lib\site-packages\" in python._pth, python.exe dies.
If I put "#Lib\site-packages\", idlexlib is said "unable to located".

Could it be a python-3.6.0b2 bug, or just a wrong-doing from WinPython ?

"Python._pth"=
.
Lib
import site
DLLs
#Lib/site-packages
#python36.zip
msg279342 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-10-24 21:53
Might have to use a backslash in the path - I don't remember whether I tried to handle forward slashes or not, but I suspect not. Definitely tried it with site-packages though.
msg279354 - (view) Author: Big Stone (Big Stone) Date: 2016-10-25 05:23
python.exe crashes when I try this python._pth:
.
Lib
import site
DLLs
Lib\site-packages
#python36.zip

all variation I try on Lib\site-packages do fail, when I not commenting # the line... 
Nevertheless, jupyter/numpy/bokeh do work with original setting.

I'm lost in thoughts: how adding a line in this file can make Python crash ?
msg279423 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-10-25 17:29
Can you tell me more about the crash? It doesn't cause a crash when I try it with my own install.
msg279425 - (view) Author: Big Stone (Big Stone) Date: 2016-10-25 17:43
I just see a windows screen poping up with (translated from french)

"Python has stopped to work" 

a problem caused this program to stop working correctly. Windows is going to close this program and will inform you if a solution is available.
msg279431 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-10-25 17:58
Check in the Event Log viewer to see if there is an "Application Error" entry for python.exe. Also, if you run python.exe from a command prompt there may be more information displayed in the output.
msg279459 - (view) Author: Big Stone (Big Stone) Date: 2016-10-25 21:38
possible particularities of my PC vs yours:
- I have no python entry at all in the regex
- I have no py.exe,
- I have no Visual Studio (but the compiler)

with Windows 10, I don't know where is the even viewer.
msg279468 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-10-26 00:15
If you right-click the Start button, Event Viewer is near the top.
msg279487 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-10-26 03:23
I installed "WinPython-64bit-3.6.0.0Zerorc2.exe" on Windows 10. As you can see below, the included version of IDLEX depends on idlelib implementation details that have changed between 3.5 and 3.6:

    C:\WinPython36\python-3.6.0b2.amd64>.\python
    Python 3.6.0b2 (default, Oct 10 2016, 21:15:32) [MSC v.1900 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>>  import idlexlib
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "C:\WinPython36\python-3.6.0b2.amd64\Lib\site-packages\idlexlib\__init__.py", line 10, in <module>
        from .idlexMain import version as __version__
      File "C:\WinPython36\python-3.6.0b2.amd64\Lib\site-packages\idlexlib\idlexMain.py", line 46, in <module>
        from idlexlib.extensionManager import extensionManager
      File "C:\WinPython36\python-3.6.0b2.amd64\Lib\site-packages\idlexlib\extensionManager.py", line 60, in <module>
        from idlelib.configHandler import idleConf, IdleConfParser
    ModuleNotFoundError: No module named 'idlelib.configHandler'

The "Unable to located" [sic] error is from the idlex.py script due to the above import error.

I couldn't reproduce the crash due to python._pth. Your Windows application log should provide the DLL (module) and exception code for the crash, but a dump file would be even better. With the error reporting dialog still open, look for the crashed python.exe in the task manager details tab (the working set of the crashed process should be small -- about 100K). Right-click it and select the option to create a dump file. Zip the dump file and upload it here.
msg279503 - (view) Author: Big Stone (Big Stone) Date: 2016-10-26 17:41
Event Viewer says, when I put "Lub\site-packages" in python._pth:
""""
Nom de l’application défaillante python.exe, version : 3.6.112.1013, horodatage : 0x57fc0593
Nom du module défaillant : ucrtbase.dll, version : 10.0.14393.0, horodatage : 0x578997b5
Code d’exception : 0xc0000409
Décalage d’erreur : 0x000000000006d5b8
ID du processus défaillant : 0x190c
Heure de début de l’application défaillante : 0x01d22fafd622ed09
Chemin d’accès de l’application défaillante : C:\WinPython\basedir36\build\winpython-64bit-3.6.x.0\python-3.6.0b2.amd64\python.exe
Chemin d’accès du module défaillant: C:\WINDOWS\System32\ucrtbase.dll
ID de rapport : ec44b511-6196-48a0-95ec-dc997b4d0302
Nom complet du package défaillant : 
ID de l’application relative au package défaillant :
msg279504 - (view) Author: Big Stone (Big Stone) Date: 2016-10-26 17:53
Thanks Eryk,

So the root cause is that IDLEX is no more compatible with IDLE in python3.6.

==> I can survive this loss... Now, I don't if the "python._pth" crash is a problem, as I can stay with "#Lib\site-packages" for now.
msg279515 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-10-26 21:54
I don't know anything about 'python._pth' and whether there is any bug with respect to that.

As far as IDLE goes, there is no bug and this issue should be closed.  

In #24225, before the release of 3.6.0a2, most file names within idlelib were changed to shorter or lowercased names in conformance with PEP 8 and as anticipated by PEP 434.  In particular, 'configHandler.py' is now 'config.py'.  API changes within and between files are and will be much more disruptive to external users.  As Nick Coughlin said in msg266409, 3rd party idlelib users are free to bundle or depend on a frozen copy of a past version.

WinPython should test the third party modules it includes with the python it is releasing.  Importing a module is as minimal as it gets.  Consider reporting the incompatibility to them.
msg279527 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-10-27 06:36
You may have uncovered a bug in Python that's causing the invalid parameter handler to be invoked. It would help if you uploaded the zipped dump file for the crashed process. 

The status code you're getting (i.e. STATUS_STACK_BUFFER_OVERRUN, 0xC0000409) is used by the __fastfail intrinsic [1]. The exception occurred in ucrtbase.dll at offset 0x6d5b8. In a debugger you can see it's in the CRT's invoke_watson function:

    0:000> u ucrtbase + 0x6d5b8 - 0x18 L7
    ucrtbase!invoke_watson:
    00007ffd`8984d5a0 4883ec28        sub     rsp,28h
    00007ffd`8984d5a4 b917000000      mov     ecx,17h
    00007ffd`8984d5a9 ff1531310400    call    qword ptr
        [ucrtbase!_imp_IsProcessorFeaturePresent (00007ffd`898906e0)]
    00007ffd`8984d5af 85c0            test    eax,eax
    00007ffd`8984d5b1 7407            je      ucrtbase!invoke_watson+0x1a
                                                (00007ffd`8984d5ba)
    00007ffd`8984d5b3 b905000000      mov     ecx,5
    00007ffd`8984d5b8 cd29            int     29h

A fast fail executes an int 29h software interrupt, which gets handled by the following system function:

    lkd> !idt 0x29
    Dumping IDT: fffff801819a8070
    29:     fffff8017fd66680 nt!KiRaiseSecurityCheckFailure

In this case the CRT passes a value of 5 (FAST_FAIL_INVALID_ARG) in register rcx (ecx), so it's not a critical security failure.

[1]: https://msdn.microsoft.com/en-us/library/dn774154.aspx
msg279539 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-10-27 14:57
I suspect there's a .pth file in site-packages that is importing something to trigger the failure. Without a crash dump (or debug build) it's going to be difficult to find it, but it is certainly an unwrapped invalid parameter termination.

To save me some time, where can I get that build of WinPython from?
msg279551 - (view) Author: Big Stone (Big Stone) Date: 2016-10-27 17:28
hi Steve,

You can grab it there https://sourceforge.net/projects/winpython/files/WinPython_3.6/3.6.0.0/betas/WinPython-64bit-3.6.0.0Zerorc2.exe/download
 MD5                             | SHA-1                                    | SHA-256                                                          | Binary                          | Size              
---------------------------------|------------------------------------------|------------------------------------------------------------------|---------------------------------|-------------------
dd946ed17ee86ea035361d2e757a1cc1 | f0ec7ffac477a220dd24aea3fb70afaba579df00 | af6536f1922a044ac74300efcd275c9e25c5eb56140ded84a99f11d38ae5ac7b | WinPython-64bit-3.6.0.0Zerorc2.exe |   24 196 083 Bytes
dabae69ad09e1646625d3a8995a75056 | aaece4907096422c1df78f80a42a2268369d7697 | 0d530b84f29481e7f03e4615c9da489711ca6883b49996219d9cd13aa5393330 | WinPython-64bit-3.6.0.0rc2.exe  |  208 255 944 Bytes
msg279552 - (view) Author: Big Stone (Big Stone) Date: 2016-10-27 17:34
maybe click on the "WinPython Command Prompt.exe" and do "pip uninstall IDLEX" as a first step. so you should see IDLE working
msg279563 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-10-27 19:02
It's a genuine bug in path processing, specifically how we handle buffer resizing. I'll make a fix.
msg279571 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-10-27 21:29
New changeset eea669163131 by Steve Dower in branch '3.6':
Issue #28522: Fixes mishandled buffer reallocation in getpathp.c
https://hg.python.org/cpython/rev/eea669163131

New changeset 72e64fc8746b by Steve Dower in branch 'default':
Issue #28522: Fixes mishandled buffer reallocation in getpathp.c
https://hg.python.org/cpython/rev/72e64fc8746b
msg279572 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-10-27 21:31
Fixed and added a test. (Yes I know that it's not the most efficient algorithm for joining the strings together, but I consider correctness to be more important here.)
msg279616 - (view) Author: Big Stone (Big Stone) Date: 2016-10-28 17:06
will it be in python-3.6.0b3 ?

what should be in python._pth, in WinPython particular case ?
(as Lib\site-packages didn't seem needed, for unknown reason)
msg279617 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-10-28 17:14
Yes, see the commit to branch 3.6, which will next be released as .0b3.
msg279624 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-10-28 18:28
"Lib\site-packages" is probably unnecessary because of "import site", which likely adds it in anyway.

It's very likely that WinPython doesn't actually want to specify this at all, since it also enables isolated mode, which will ignore PYTHONPATH and the current working directory. But if that's okay, you probably want::

    python36.zip
    DLLs
    Lib
    .
    import site

That should give you the same default sys.path as if the file were omitted, except for the empty entry at the start and anything in PYTHONPATH or the registry. (I figured this out by running "python -S" and looking at sys.path.)
msg279626 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-10-28 20:42
> ignore PYTHONPATH and the current working directory

Generally it's the script directory that isolated mode removes from sys.path. It's the working directory when there is no script.
msg279690 - (view) Author: Big Stone (Big Stone) Date: 2016-10-29 18:18
thank you all for the patch

IDLEX was a requirement for a french examination.

I think the reason was to see the line numbers on the left of the editor.

For sure since IDLEX birth, IDLE has made some progress and IDLEX is becoming irrelevant, but this lovely tiny feature seems still missing.
msg279691 - (view) Author: Big Stone (Big Stone) Date: 2016-10-29 18:29
the "show line number on the left" feature is on the "github web editor", on "atom", and on "spyder" and "erik" python IDE, so rather the expected standard for python editing.
msg279703 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-10-30 02:57
#17535 is about adding line numbers to IDLE editor.  I have approved it in priciple, but not yet the proposed patch, or a revision thereof.
msg279871 - (view) Author: Big Stone (Big Stone) Date: 2016-11-01 11:22
the suggested python._pth change makes python unhappy:

    python36.zip
    DLLs
    Lib
    .
    import site

Récipient d’erreurs 116251549737, type 5
Nom d’événement : BEX64
Réponse : Non disponible
ID de CAB : 0

Signature du problème : 
P1 : python.exe
P2 : 3.6.112.1013
P3 : 57fc0593
P4 : ucrtbase.dll
P5 : 10.0.14393.0
P6 : 578997b5
P7 : 000000000006d5b8
P8 : c0000409
P9 : 0000000000000005
P10 : 

Fichiers joints :
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERF455.tmp.WERInternalMetadata.xml
\\?\C:\Users\famille\AppData\Local\Temp\WERFF72.tmp.appcompat.txt

Ces fichiers sont peut-être disponibles ici :
C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_python.exe_ec9ba1cd21c08ca3de75f183bf945ce737867af_b3b3e14f_1575008a

Symbole d’analyse : 
Nouvelle recherche de la solution : 0
ID de rapport : 66625eb0-e311-4368-be37-4b600bcb8978
Statut du rapport : 1
Récipient avec hachage : 57d09a5cb4a63ab86af1e62bd270b573
msg279872 - (view) Author: Big Stone (Big Stone) Date: 2016-11-01 11:24
oups! I may have test the old one...
msg279873 - (view) Author: Big Stone (Big Stone) Date: 2016-11-01 11:26
it looks ok with 3.6.0b3 ... sorry for the false alarm
History
Date User Action Args
2017-03-31 16:36:32dstufftsetpull_requests: + pull_request1052
2017-03-20 09:54:33illagrenansetnosy: + illagrenan
2016-11-01 11:26:58Big Stonesetmessages: + msg279873
2016-11-01 11:24:37Big Stonesetmessages: + msg279872
2016-11-01 11:22:39Big Stonesetfiles: + Report.wer

messages: + msg279871
2016-10-30 02:57:03terry.reedysetmessages: + msg279703
2016-10-29 18:29:03Big Stonesetmessages: + msg279691
2016-10-29 18:18:23Big Stonesetmessages: + msg279690
2016-10-28 20:42:54eryksunsetmessages: + msg279626
2016-10-28 18:28:03steve.dowersetstatus: open -> closed
resolution: fixed
messages: + msg279624

stage: commit review -> resolved
2016-10-28 17:14:10terry.reedysetmessages: + msg279617
2016-10-28 17:06:52Big Stonesetmessages: + msg279616
2016-10-27 21:31:35steve.dowersettype: crash
stage: needs patch -> commit review
messages: + msg279572
versions: + Python 3.7
2016-10-27 21:29:38python-devsetnosy: + python-dev
messages: + msg279571
2016-10-27 19:02:17steve.dowersetassignee: steve.dower
messages: + msg279563
stage: needs patch
2016-10-27 17:34:55Big Stonesetmessages: + msg279552
2016-10-27 17:28:07Big Stonesetmessages: + msg279551
2016-10-27 14:57:58steve.dowersetmessages: + msg279539
2016-10-27 06:36:22eryksunsetmessages: + msg279527
2016-10-26 21:54:14terry.reedysetmessages: + msg279515
2016-10-26 19:46:26ned.deilysetnosy: + terry.reedy, roger.serwy
2016-10-26 17:53:16Big Stonesetmessages: + msg279504
2016-10-26 17:41:16Big Stonesetmessages: + msg279503
2016-10-26 03:23:37eryksunsetnosy: + eryksun
messages: + msg279487
2016-10-26 00:15:38steve.dowersetmessages: + msg279468
2016-10-25 21:38:47Big Stonesetmessages: + msg279459
2016-10-25 17:58:15steve.dowersetmessages: + msg279431
2016-10-25 17:43:03Big Stonesetmessages: + msg279425
2016-10-25 17:29:38steve.dowersetmessages: + msg279423
2016-10-25 05:23:24Big Stonesetmessages: + msg279354
2016-10-24 21:53:56steve.dowersetmessages: + msg279342
2016-10-24 20:45:13Big Stonecreate