This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author Thrameos
Recipients Thrameos, paul.moore, steve.dower, tim.golden, zach.ware
Date 2020-12-16.15:25:48
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1608132349.09.0.350852588073.issue42529@roundup.psfhosted.org>
In-reply-to
Content
I am fairly sure this is a Python "bug" in the sense that there was some change in undocumented change in requirements and the distutils pattern for building a module no longer reflects that requirement.   That said very likely JPype is the only module to be affected and thus I will have to manually adjust to account for the new requirement.

Unfortunately as far as developers, I am it so fixing it (with your assistance) is going to have to fall on me.  So lets do a run down of how this all working so you can point me where to look.

1) JPype gets built as an ordinary setup.py CPython module.  So it is possibly a bug in the build pattern of the customizers in JPype that was exposed by Python 3.9.   I am just going to cut and paste so that it is easy to follow along.

jpype/setup.py
```
...
from setuptools import Extension
...
import setupext
...
jpypeLib = Extension(name='_jpype', **setupext.platform.Platform(
    include_dirs=[Path('native', 'common', 'include'),
                  Path('native', 'python', 'include'),
                  Path('native', 'embedded', 'include')],
    sources=[Path('native', 'common', '*.cpp'),
             Path('native', 'python', '*.cpp'),
             Path('native', 'embedded', '*.cpp')], platform=platform,
))
```

We have two sets of customization in setup.py.  Both are included from a module called jpype/setupext/

The key one is the jpype/setupext/platform.py which defines the compiler flags.  There are two sections of interest...

jpype/setupext/platform.py contains these modifications for win32.   
(So there is the Advapi32, not sure why it appears there because this section is all before my time as this was started in 2001 and I joined in 2018)
```
    static = True
    if platform == 'win32':
        distutils.log.info("Add windows settings")
        platform_specific['libraries'] = ['Advapi32']
        platform_specific['define_macros'] = [('WIN32', 1)]
        if sys.version > '3':
            platform_specific['extra_compile_args'] = [
                '/Zi', '/EHsc', '/std:c++14']
        else:
            platform_specific['extra_compile_args'] = ['/Zi', '/EHsc']
        platform_specific['extra_link_args'] = ['/DEBUG']
        jni_md_platform = 'win32'
```

The second section is currently commented out.  Though it is relevant because JPype is exotic in one way.  It is a module which is loaded in three ways.   When imported from Python it is an ordinary library (1) which will later pull in Java which will then load library a second time (2) to bind Java native methods.   It can also be used to launch Python if Java starts the session (3).   In that case, it needs libpython.dll to be bound to module so that the Java equivalent to LoadLibrary can work.  When it does Java first manually loads libpython.dll then loads the module and calls the hook to get Python started. 
```
# This code is used to include python library in the build when starting Python from
# within Java.  It will be used in the future, but is not currently required.
#    if static and sysconfig.get_config_var('BLDLIBRARY') is not None:
#        platform_specific['extra_link_args'].append(sysconfig.get_config_var('BLDLIBRARY'))
```

The actual buildext has a few minor patches so that Java libraries can run through the normal process.  But nothing seems like a good candidate

We have one section tweeking some of the build options.
```
    def initialize_options(self, *args):
        """omit -Wstrict-prototypes from CFLAGS since its only valid for C code."""
        self.android = False
        self.makefile = False
        self.jar = False
        import distutils.sysconfig
        cfg_vars = distutils.sysconfig.get_config_vars()
        replacement = {
            '-Wstrict-prototypes': '',
            '-Wimplicit-function-declaration': '',
        }
        tracing = self.distribution.enable_tracing

        # Arguments to remove so we set debugging and optimization level
        remove_args = ['-O0', '-O1', '-O2', '-O3', '-g']

        for k, v in cfg_vars.items():
            if not isinstance(v, str):
                continue
            if not k == "OPT" and not "FLAGS" in k:
                continue

            args = v.split()
            for r in remove_args:
                args = list(filter((r).__ne__, args))

            cfg_vars[k] = " ".join(args)
        super().initialize_options()
```

Then later we interrupt the build process for Java.
```
    def build_extension(self, ext):
        if ext.language == "java":
            return self.build_java_ext(ext)
        if self.jar:
            return
        print("Call build ext")
        return super().build_extension(ext)
```

2) Next we have the module start.  I am guessing this is not the source of the error because adding a printf shows we never got to this point.

jpype/native/pyjp_module.cpp
```
PyMODINIT_FUNC PyInit__jpype()
{
        JP_PY_TRY("PyInit__jpype");
        JPContext_global = new JPContext();
        // This is required for python versions prior to 3.7.
        // It is called by the python initialization starting from 3.7,
        // but is safe to call afterwards.
        PyEval_InitThreads();

        // Initialize the module (depends on python version)
        PyObject* module = PyModule_Create(&moduledef);
        Py_INCREF(module);
        PyJPModule = module;
        PyModule_AddStringConstant(module, "__version__", "1.2.1_dev0");
         ... initialize a bunch of resources
     }

 ```

3) After this point we will start getting exotic.   JPype plugs into Python in with less that above board methods.  In order to operate it needs to derive a bunch of Python basic types in ways that the API does not allow. As Python does not support true multiple inheritance of concrete types and I need to add slots that should not exist, it will use a custom memory allocator to offset the addresses of object pointers such that a hidden Java slot appears on types type, object, long, double, exception, etc.  To do this it will forward declare some of the private Python symbols having to do with heap types and stacktrace frames, etc.   There is also one monkey patch for the "function" type in which it temporarily disables the "final" field so that we create a derived class that should not exist for Java methods, then restores the field immediately after.

This is relevant because if the symbols it is pulling in are missing for some reason then the dll will fail to load.   However, given it will load when it isn't compiled and not load when compiled there is not much chance the symbol table should be different.   Unless windows is really different from other architectures, this is a long shot.

3) Okay so on to module loading

```
import _jpype
```

This statement works when the debugger is attached, when called from a .py file but fails when called from a .pyc file.   I have looked at it all the way down to the byte code.   There appears to be nothing in JPype that is going into that process.  This just jumps straight into Python core code.

4) Post load a whole lot of stuff happens. We patch in the new types, and then use LoadLibrary to pull in jvm.dll.  As we work on many JVM versions and we won't know which is requested, JPype is not actually compiled against the jvm.  Instead it manually loads all its symbols using  GetProcAddress in jpype/native/jp_platform.cpp.

So that is all there is (dirty laundary and all).   Does anything look suspect?
History
Date User Action Args
2020-12-16 15:25:49Thrameossetrecipients: + Thrameos, paul.moore, tim.golden, zach.ware, steve.dower
2020-12-16 15:25:49Thrameossetmessageid: <1608132349.09.0.350852588073.issue42529@roundup.psfhosted.org>
2020-12-16 15:25:49Thrameoslinkissue42529 messages
2020-12-16 15:25:48Thrameoscreate