classification
Title: Document why embedding the interpreter can sometimes crash.
Type: Stage: resolved
Components: Documentation, Windows Versions: Python 3.7, Python 3.6, Python 3.5
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Decorater, steve.dower, tim.golden
Priority: normal Keywords:

Created on 2017-06-05 13:47 by Decorater, last changed 2017-06-05 19:58 by paul.moore. This issue is now closed.

Messages (13)
msg295187 - (view) Author: Decorater (Decorater) * Date: 2017-06-05 13:47
According to this example at https://docs.python.org/3/extending/embedding.html#very-high-level-embedding you can use something similar on embedding the interpreter.

However for some reason it can crash on Windows and probably some other platforms no matter what python code you enter in the string. I am currently getting that issue on Windows even with the python DLL's being in the same folder of the embedded executable it seems (when I open the crash dump in VS2010 express) that it tried to access an memory address that it has no permissions to.

However I have been thinking as to why that might be. The system should have plenty of memory left for it to allocate and run just fine unless it wants to use an already active python process (why would that be when other applications that embed the interpreter seem to work just fine).

I think documenting there on when and iff the compiled code crashes with suggestions on how to fix them would be best to do on the page.

it could be that the example actually compiles but does not work anymore too. But why would it be on the page if it would compile but crash on any string put into it? Something makes me think the crash is on the Py_DecodeLocale line with the size_t * param being NULL.
msg295188 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2017-06-05 13:55
I'm a bit confused what you're asking for here.

Certainly if you embed Python, you can get crashes, just as with any other C program. Pass an invalid pointer to a Python API, build your application and the Python DLL with incompatible C runtimes, free memory you don't own, ...

That's not really something we need to cover in the Python documentation, though, it's just things you need to understand if you're doing anything beyond the most trivial C programming.

If there's a specific issue that you're having, we'd need more details to say anything else.
msg295190 - (view) Author: Decorater (Decorater) * Date: 2017-06-05 14:22
Well the example I linked if I compile it and try to run it it instantly crashes with no explainable reason. I am using the same compiler used to build the core too and the normal interpreter works just fine with the code I am trying to embed. So it seems to definitely be the Py_DecodeLocale or the Py_SetProgramName line.
msg295191 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2017-06-05 14:31
Worked fine for me:

>cl .\pyc.c /I $env:LOCALAPPDATA\Programs\Python\Python36\include $env:LOCALAPPDATA\Programs\Python\Python36\libs\python36.lib
Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24213.1 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

pyc.c
Microsoft (R) Incremental Linker Version 14.00.24213.1
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:pyc.exe
pyc.obj
...\Programs\Python\Python36\libs\python36.lib

>.\pyc.exe
Today is Mon Jun  5 15:27:45 2017

So it doesn't look like the problem is with the code, so much as with how you've done the build. The problem here is that there are so many possibilities for C compilation environments, that there's simply no way we could give a "this is how you set up to build an embedding application".

If you have specific suggestions for how to improve the documentation, a patch would be welcome. But in the absence of specifics, I don't think there's anything actionable in this report.
msg295193 - (view) Author: Decorater (Decorater) * Date: 2017-06-05 14:39
hmm, maybe it was because I was compiling it inside of the VS2015 IDE in an project file and not with cl.exe directly.
msg295194 - (view) Author: Decorater (Decorater) * Date: 2017-06-05 14:48
Well looks like with cl.exe it is still crashing.
msg295195 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2017-06-05 14:53
My bet is still on something like you compiling your code 32-bit and linking to 64-bit Python, or something like that.
msg295196 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-06-05 14:54
Agreed, there's nothing for us to do here.  It sounds like you may need to go to a help forum to work through your crash.

If you learn things that you think it would be valuable to add to the docs, please do submit a doc enhancement issue/PR.
msg295197 - (view) Author: Decorater (Decorater) * Date: 2017-06-05 14:56
I am compiling and linking to 32 bit python based on the folder I entered for the lib file. Unless the lib file in the lib folder is somehow for the 64 bit one.
msg295198 - (view) Author: Decorater (Decorater) * Date: 2017-06-05 15:12
Wait a minute, I think it is because the 32 and 64 bit python36.lib files are in the path environment variable so that might be the reason for the crash.
msg295199 - (view) Author: Decorater (Decorater) * Date: 2017-06-05 15:19
E:\Users\Elsword\Documents\GitHub\TinyURL\app>call "E:\Program Files (x86)\Micro
soft Visual Studio 14.0\Common7\Tools\VsDevCmd.bat"

E:\Users\Elsword\Documents\GitHub\TinyURL\app>cd "E:\Users\Elsword\Documents\Git
Hub\TinyURL\app\"

E:\Users\Elsword\Documents\GitHub\TinyURL\app>RC.exe /r TinyURL.rc
Microsoft (R) Windows (R) Resource Compiler Version 10.0.10011.16384
Copyright (C) Microsoft Corporation.  All rights reserved.


E:\Users\Elsword\Documents\GitHub\TinyURL\app>cl.exe /Tc "TinyURL.c" /I E:\Python360\include E:\Python360\libs\python36.lib /link /OUT:"TinyURL.exe" /SUBSYSTEM:CONSOLE "TinyURL.res" /FIXED
Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24215.1 for x86
Copyright (C) Microsoft Corporation.  All rights reserved.

TinyURL.c
TinyURL.c(204): warning C4129: '/': unrecognized character escape sequence
Microsoft (R) Incremental Linker Version 14.00.24215.1
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:TinyURL.exe
/OUT:TinyURL.exe
/SUBSYSTEM:CONSOLE
TinyURL.res
/FIXED
TinyURL.obj
E:\Python360\libs\python36.lib

Wait a minute, it seems the build tools are a bit out of date.
msg295202 - (view) Author: Decorater (Decorater) * Date: 2017-06-05 16:26
I found where the crash was at. It was at my call to PyRun_SimpleString.

Something in the string caused it to crash.
msg295204 - (view) Author: Decorater (Decorater) * Date: 2017-06-05 16:45
turns out it was crashing because I forgot to add in Py_Initialize.
History
Date User Action Args
2017-06-05 19:58:07paul.mooresetnosy: - paul.moore
2017-06-05 19:39:47r.david.murraysetnosy: - r.david.murray
2017-06-05 16:45:31Decoratersetmessages: + msg295204
2017-06-05 16:26:18Decoratersetmessages: + msg295202
2017-06-05 15:25:56zach.waresetassignee: docs@python ->

nosy: - docs@python, zach.ware
2017-06-05 15:19:25Decoratersetmessages: + msg295199
2017-06-05 15:12:17Decoratersetmessages: + msg295198
2017-06-05 14:56:38Decoratersetmessages: + msg295197
2017-06-05 14:54:15r.david.murraysetstatus: open -> closed

nosy: + r.david.murray
messages: + msg295196

resolution: not a bug
stage: resolved
2017-06-05 14:53:28paul.mooresetmessages: + msg295195
2017-06-05 14:48:47Decoratersetmessages: + msg295194
2017-06-05 14:39:47Decoratersetmessages: + msg295193
2017-06-05 14:31:57paul.mooresetmessages: + msg295191
2017-06-05 14:22:56Decoratersetmessages: + msg295190
2017-06-05 13:55:46paul.mooresetmessages: + msg295188
2017-06-05 13:47:58Decoratercreate