classification
Title: Running extension modules using -m switch
Type: enhancement Stage:
Components: Extension Modules Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Dormouse759, encukou, ncoghlan, terry.reedy
Priority: normal Keywords:

Created on 2017-05-19 11:20 by Dormouse759, last changed 2017-05-23 13:54 by Dormouse759.

Pull Requests
URL Status Linked Edit
PR 1761 open Dormouse759, 2017-05-23 13:54
Messages (6)
msg293954 - (view) Author: Marcel Plch (Dormouse759) * Date: 2017-05-19 11:20
Currently the -m switch does not work with extension modules:
    
    $ python3 -m math

    /usr/bin/python3: No code object available for math


In order to enable extension modules to behave like Python source modules,
the -m switch should be supported.

Please, see this proof of concept:
https://github.com/Traceur759/cpython/tree/main_c_modules
msg293979 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-05-19 21:43
What is the use case?

It only make sense to run any stdlib module with -m, and without -i, if it has a command line interface (and an if __name__ clause).  Otherwise, the module is created and then deleted when python exits.

> py -m math
>

C-coded modules with such a command line interface have a mod.py file that imports _mod.

The following can make sense.

> py -m -i math  
Python x.y ...
>>> math.sin(1.48973)

But it is hardly needed as '-m -i math' is only one char less than 'import math'
msg293983 - (view) Author: Petr Viktorin (encukou) * Date: 2017-05-19 23:17
It's part of a larger effort to bring the capabilities of extension modules up to par with Python ones. For example, it's one less surprise you'd get when you Cythonize a module. And it's not only for stdlib modules – it's for any extension module.



I'd be happy to answer questions here. But to get up to speed and avoid bpo comment lag, you can also see the current discussion on import-sig [0], or dig up some of the older conversations there leading up to PEP 489 (Feb-May 2015). And if you're at PyCon, you could have a high-bandwidth conversation with Nick Coghlan – he has the master plan in his head :)

[0] https://mail.python.org/pipermail/import-sig/2017-May/001072.html
msg294016 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-05-20 07:22
As a high level overview of the general idea: we'd like it to be almost entirely transparent to the end user as to whether a particular module is implemented as normal Python source code, a precompiled bytecode/wordcode file, or a precompiled Cython extension module (or equivalent).

At the moment, this is pretty close to being true for source code vs precompiled bytecode/wordcode when it comes to both imports and execution as a script. The main missing piece there is to implement source code maps for generating more informative tracebacks given only the precompiled form (perhaps by borrowing JavaScript's "source map" concept)

For extension modules, the original multi-phase initialisation PEP got this pretty close to being true for the import case - things like reload() can now work much the same way they do for pure Python and pyc files if a module author (or module generation tool) cares to make it so.

However, we don't yet support the use of extension modules as scripts, neither for direct execution, nor via the `-m` switch, so it's impossible for a tool like Cython to handle that transparently - if a module exposes functionality via -m, then migrating it directly to Cython will break than behaviour.
msg294019 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-05-20 08:13
So a use case might be that someone could compile all the stdlib .py modules with cython, as they are, without touching the code, and have the result be a drop-in replacement.  I'd like that to be possible.
msg294032 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-05-20 14:37
Yep, that's the kind of thing we'd like to make possible.
History
Date User Action Args
2017-05-23 13:54:58Dormouse759setpull_requests: + pull_request1845
2017-05-20 14:37:48ncoghlansetmessages: + msg294032
2017-05-20 08:13:01terry.reedysetmessages: + msg294019
2017-05-20 07:22:43ncoghlansetmessages: + msg294016
2017-05-19 23:17:53encukousetmessages: + msg293983
2017-05-19 21:43:52terry.reedysetnosy: + terry.reedy
messages: + msg293979
2017-05-19 12:01:30encukousetnosy: + ncoghlan, encukou
2017-05-19 11:20:42Dormouse759create