This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Embbedable zip does not import modules in (zip)subdirectory
Type: behavior Stage: resolved
Components: Interpreter Core, Windows Versions: Python 3.5
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: SimonG, amaury.forgeotdarc, belopolsky, meador.inge, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2016-03-21 05:05 by SimonG, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
embtest.c zach.ware, 2016-03-22 04:34
Messages (12)
msg262101 - (view) Author: Simon (SimonG) Date: 2016-03-21 05:05
I downloaded the Embeddable zip file from the Python site, (https://www.python.org/downloads/release/python-351/) and rand the code below.

When I run the code below certain modules are not imported while others are imported fine.
The modules that are not imported seem to be the ones sub-directories within the zip file.

(omitted obvious code, error checking and some cleanup)

...
    int main()
    {
      // ... 
      Py_SetProgramName(L"AAAAA"); 
      Py_SetPath(L"path\\to\\python35.zip"); // embeddable from the python site. 
      Py_Initialize(); 
      PyThreadState *mainstate = PyThreadState_Get(); 
      PyInterpreterState* mainInterpreterState = mainstate->interp; 
      
      PyObject *main_module = PyImport_AddModule("__main__"); 
      PyObject *main_dict = PyModule_GetDict(main_module); 
      PyObject *local_dic = PyDict_New(); 
    
      const char* s = "import ctypes\n"; 
      PyObject * PyRes = PyRun_String(s, Py_file_input, main_dict, local_dic); 
      PyObject* ex = PyErr_Occurred(); 
      if (NULL != ex) 
      { 
        //  didn't work, import error. 
      } 
    }

The exact same code above manages to import other modules as long as they are located inside the root of the zip file.

modules like json/ctypes and so on are not imported as they are in a sub directory.
msg262147 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-03-21 20:27
I can't reproduce this - I used almost exactly your code and it worked fine.

The one change I made was to use either L"python35.zip" alone (as my exe was alongside the zip) or a full path to the zip file. Both of these worked.

If the zip file is not found, you'll get an error. Is it possible that you are not actually pointing at the correct zip file?
msg262158 - (view) Author: Simon (SimonG) Date: 2016-03-22 03:50
I have tried with the zip file in both the exe alongside the zip and not alongside it, (mine was not).

And I get the same error in both cases.

I am using Windows 10, VS2015.

I compile the Python35.dll directly and my config is default, I don't change anything really.

Do you build the Python35.dll or where do you get it from? Do you add something to your config?
msg262159 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-03-22 04:34
I am also unable to reproduce.  I've attached the exact code I used, and built it with:

cl /I "C:\Program Files\Python35\include" embtest.c /link /libpath:"C:\Program Files\Python35\libs"

Output is the expected result of 'import ctypes;print(dir(ctypes))'.

Simon: If you run this code (including PyErr_Print();), what output do you get?  Are you sure that _ctypes.pyd is also available?  What happens if you import unittest instead of ctypes?  Or winsound?

Steve: I notice that the embeddable zip does not include 'include' or 'libs' dirs, was this deliberate?  I just noticed because it caused me to jump through an extra hoop to test this after I mistakenly downloaded the 32-bit embeddable zip on the first try, then had to go back for the 64-bit (or a 32-bit installer).  If it's deliberate (and preferably documented somewhere that I didn't check for), that's fine; I'm just curious.
msg262160 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-03-22 04:39
I used the python35.dll in the embeddable install, along with the python35.zip included in there. I linked against python.h and python35.lib from a full install, but put the resulting exe in the directory with the embeddable install so it uses that instead of the main one.

Probably I ought to write up a more detailed walkthrough on how to customise the embeddable install, but your feedback here is very valuable. Did you do things differently from what I put above? You're certainly not the only one to do whatever you did.
msg262163 - (view) Author: Simon (SimonG) Date: 2016-03-22 06:07
1- New solution Win32 "console application", (left all default settings).
2- downloaded "Gzipped source tarball" from https://www.python.org/downloads/release/python-351/
3- Extracted everything, (didn't change anything).
4- Added "pythoncore.vcxproj" in "Python-3.5.1\PCbuild" to the solution above.
5- Compiled to make sure... "python35_d.dll" and "python35_d.lib" created in "Python-3.5.1\PCbuild\win32"
6- Added the "Python-3.5.1\PCbuild\win32" to my library include folder
7- Added "Python-3.5.1\Include" to my additional include folder. (for Python.h)
7- Added "Python-3.5.1\PC" to my additional include folder. (for Pyconfig.h)
8- Added "#include "Python.h"" to my ConsoleApplicaiton.cpp
9- Compile, all good.
10- downloaded "Windows x86-64 embeddable zip file" from link above.
11- Copied the created "python35_d.dll" to my debug folder.
12- Replaced "path\\to\\python35.zip" with the full path of the embeddable zip file.

And I get the same issue.

PyErr_Print(); output.

Trackback (most recent call last):
  File "<string>", line 1, in <module>
  File "ctypes\__init__.py, line 8, in <module>
ImportError: No module named '_ctypes'

"import unittest" - works

"import winsound" - does not work, same error as above.

Please let me know if I should do anything else.
msg262164 - (view) Author: Simon (SimonG) Date: 2016-03-22 06:15
Sorry I should add that 

1- I do *not* have python installed on my dev machine.
2- VS2015 Enterprise.
3- Windows 10 - x64
4- *Not* running as admin

Everything else is fairly standard.
msg262182 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-03-22 13:18
Are you making a debug build? It might be looking for _ctypes_d.pyd in that case. You'll need a release build for the embeddable distro, as well as including the pyds from the zip in sys.path - they need to be standalone files.

The embeddable release is meant only for redistribution, so you should build and test against a full install (including debug binaries if you want) and do integration testing with the embeddable package and a release build. (So Zach, yes, it's deliberate.)
msg262186 - (view) Author: Simon (SimonG) Date: 2016-03-22 14:47
Yes, it was a debug build, I didn't know it only works in release, that's the part I was clearly missing.

It would be great if we could have a debug embeddable zip file, but I understand that it might be asking for a bit much.

Is the project that builds all the pyds available? Maybe I could create my own embeddable debug zip.

Thanks
msg262189 - (view) Author: Simon (SimonG) Date: 2016-03-22 15:21
Not sure if I should be closing the issue or if you should.
I think it is not an issue.
msg262194 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-03-22 16:14
Each of the .pyd files has its own .vcxproj in PCbuild.  You might be able to make use of pcbuild.sln or pcbuild.proj, each of which include all of the various project files.
msg262197 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-03-22 16:35
Unless you're working on CPython itself, I'd suggest just installing the debug binaries through the installer.

I've made a note to write a blog post on how to use the embeddable distro, so when that comes out (in a few weeks, probably) hopefully it will help answer these questions. Then I'll look at merging the parts of that which would be good documentation into the Doc/using/windows.rst page.
History
Date User Action Args
2022-04-11 14:58:28adminsetgithub: 70785
2016-03-22 16:35:20steve.dowersetmessages: + msg262197
2016-03-22 16:14:53zach.waresetstatus: open -> closed
resolution: not a bug
messages: + msg262194

stage: resolved
2016-03-22 15:21:11SimonGsetmessages: + msg262189
2016-03-22 14:47:20SimonGsetmessages: + msg262186
2016-03-22 13:18:54steve.dowersetmessages: + msg262182
2016-03-22 06:15:01SimonGsetmessages: + msg262164
2016-03-22 06:07:32SimonGsetmessages: + msg262163
2016-03-22 04:39:13steve.dowersetmessages: + msg262160
2016-03-22 04:35:02zach.waresetfiles: + embtest.c

messages: + msg262159
2016-03-22 03:50:40SimonGsetmessages: + msg262158
2016-03-21 20:27:16steve.dowersetmessages: + msg262147
2016-03-21 06:29:35SimonGsetcomponents: + Interpreter Core, Windows
2016-03-21 05:05:16SimonGcreate