This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Py_Initialize aborts when using static Python version. Windows
Type: crash Stage: resolved
Components: Interpreter Core Versions: Python 3.8
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: illera88, ncoghlan, vstinner
Priority: normal Keywords:

Created on 2018-07-05 22:41 by illera88, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (5)
msg321140 - (view) Author: Alberto (illera88) Date: 2018-07-05 22:41
Hi,

I've followed the build instructions to get a statically linked Python library in windows. The compilation works great and I get a big fat statically linked .lib file. 

When I use it and in my code I call Py_Initialize() the program aborts and I get this error:

Fatal Python error: initfsencoding: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

It seems that python is looking for encodings on the file system instead of looking for the built-in one since if I do Py_SetPythonHome(L"C:\\Python37.0-x64"); before calling Py_Initialize it works fine. 

Why is Python looking for external modules when it is a statically linked library and encodings should be built-in?

How can I indicate Python to look for the modules in itself and not externally?

Regards
msg321156 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-07-06 07:27
> Why is Python looking for external modules when it is a statically linked library and encodings should be built-in?

Hi, this is not a bug. Python needs its standard library. You might want to put the whole stdlib into a single ZIP, but I don't recall how to do that.

The encodings and codecs modules are not built-in, but _codecs is built-in:

>>> import encodings, codecs, _codecs
>>> encodings
<module 'encodings' from '/usr/lib64/python3.6/encodings/__init__.py'>
>>> codecs
<module 'codecs' from '/usr/lib64/python3.6/codecs.py'>
>>> _codecs
<module '_codecs' (built-in)>

I suggest to close this issue as "not a bug". You should ask on a Python mailing list or forum to get help how to build a standalone Python.
msg321181 - (view) Author: Alberto (illera88) Date: 2018-07-06 16:30
Hi,

First of all thank you for your answer.
If every module can't be statically compiled within a library, what is the
point of having instructions to compile python statically in the build
instructions?

Can I modify the code to make it completely statically?

Thank you

On Fri, Jul 6, 2018 at 12:27 AM STINNER Victor <report@bugs.python.org>
wrote:

>
> STINNER Victor <vstinner@redhat.com> added the comment:
>
> > Why is Python looking for external modules when it is a statically
> linked library and encodings should be built-in?
>
> Hi, this is not a bug. Python needs its standard library. You might want
> to put the whole stdlib into a single ZIP, but I don't recall how to do
> that.
>
> The encodings and codecs modules are not built-in, but _codecs is built-in:
>
> >>> import encodings, codecs, _codecs
> >>> encodings
> <module 'encodings' from '/usr/lib64/python3.6/encodings/__init__.py'>
> >>> codecs
> <module 'codecs' from '/usr/lib64/python3.6/codecs.py'>
> >>> _codecs
> <module '_codecs' (built-in)>
>
> I suggest to close this issue as "not a bug". You should ask on a Python
> mailing list or forum to get help how to build a standalone Python.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue34057>
> _______________________________________
>
-- 
Alberto García Illera

GPG Public Key: https://goo.gl/twKUUv
msg321192 - (view) Author: Alberto (illera88) Date: 2018-07-06 21:03
Hey Victor,

I've spent another number of hours debugging the problem and trying to avoid encodings to be loaded but I've had no luck since initfsencoding is necesary for init_sys_streams to work and that's an important function that sets stdin, stout and sterr. 

It seems that encodings is the only thing that prevents Py_Initialize to be called and therefore for a static python version to be used.

Is there any way to move encodings to pythoncore? Is there any other solution? I'm happy to put my time on this (I've already spent 3 full days debugging the problem).

I think this could be considered a bug since there is no point on statically build python and use it as a library as the README stands if there is still requirements on encodings which need to reside on the filesystem. 

Thank you again for your time
msg321256 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-07-08 04:26
encodings is only the *first* unfrozen Python module that we load - once you get past that, then you'll find that we also load os.py and various other things.

cx_freeze is an illustrative example to look at in that regard, as it preconfigures the interpreter to be able to find the cx_freeze generated zip archive that has the program's Python modules in it: https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Common.c

The technique that cx_freeze doesn't use yet is to combine the statically linked Python binary and the generated zip archive into a single file (similar to what zipapp does), and adjust the sys.path definition inside the binary to refer back to the executable itself (since executable files can have arbitrary content appended, while zip files can have arbitrary content *pre*pended).

We're always going to require that embedding applications provide a working import path of some kind, so this isn't a bug.

It might be worth converting to a documentation enhancement request though, as we don't really make it clear in https://docs.python.org/3/c-api/init.html what the "minimum viable import set" actually looks like.
History
Date User Action Args
2022-04-11 14:59:02adminsetgithub: 78238
2018-07-08 04:26:50ncoghlansetstatus: open -> closed
resolution: not a bug
messages: + msg321256

stage: resolved
2018-07-08 00:09:06ncoghlansetnosy: + ncoghlan
2018-07-06 21:03:41illera88setmessages: + msg321192
2018-07-06 16:30:04illera88setmessages: + msg321181
2018-07-06 07:27:31vstinnersetmessages: + msg321156
2018-07-05 22:46:26vstinnersetnosy: + vstinner
2018-07-05 22:41:49illera88create