The -m interpreter option allows one to run library module as a script, but if you want to debug, profile or trace the execution of the same, you must supply the path to the module source file on the command line.  The resulting execution may also be different from python -m run especially when the module is located within a package.

I would like to be able to do

$ python -m trace <trace options> --run-module <module name>

and the same with pdb and profile in place of trace.

I've thought about this in the past, but never really pursued it due to the question of what to do with the __main__ namespace.

There are three options here:
1. Use runpy.run_module to run the module in a fresh __main__ namespace
2. Use runpy.run_module to run the module under its own name
3. Use runpy._run_module_as_main to run the module in the real __main__ namespace

Option 3 is probably a bad idea (due to the risk of clobbering globals from pdb/trace/profile/doctest/etc) but failing to do it that way creates a difference between the way the actual -m switch works and what these modules will be doing.

That said, I haven't looked closely at what these modules do for ordinary scripts, where much the same problem will already arise. If option 1 is adequate for this purpose, then it shouldn't be that hard to add - it's just that I've never done the investigation to see if it *would* be adequate.

I am afraid, for ordinary scripts these modules effectively use option 3.  I think these modules should remove its own scaffolding from "real" __main__ before loading any traced code.  I am not sure how this can be achieved, though.

On Wed, Oct 6, 2010 at 2:59 AM, Alexander Belopolsky
<report@bugs.python.org> wrote:
>
> Alexander Belopolsky <belopolsky@users.sourceforge.net> added the comment:
>
> I am afraid, for ordinary scripts these modules effectively use option 3.  I think these modules should remove its own scaffolding from "real" __main__ before loading any traced code.  I am not sure how this can be achieved, though.

If you use runpy.run_module or runpy.run_path, they will switch the
existing __main__ out of sys.modules, replacing it with a temporary
module. However, that approach is currently slightly broken, in that
it leaves the temporary module namespace inaccessible if the module
execution fails with an exception (hence the existence of
run_module_as_main). I've thought of a few ways to fix that, but never
explored any of them:
- allow the module to be used for execution to be passed in to
run_module and run_path as a new optional parameter
- allow a list (or other mutable container) to be passed in as an
output parameter, and stick the temporary module in there
- define a thread-local variable for the runpy module that stores the
last module namespace executed via runpy in the current thread (and a
convenience API for retrieving it)

Option 2 strikes me as rather clumsy, so we can probably skip that. I
find option 3 to be quite elegant in a sys.exc_info() kind of way, but
option 1 is probably simpler.

Following Nick's advice, I extended runpy.run_module to accept an extra parameter to be used as replacement __main__ namespace.

Having this, I can make this temporary __main__ accessible in main() in modules like trace/profile/pdb even if module execution fails with an exception. The problem is that it's visible only in the calling function but not in the global namespace. One way to make it accessible for post mortem debugging would be to create the replacement __main__ module in the global namespace and then pass as a parameter to main(), but this seems clumsy.

So maybe the way to go is to have runpy store last used __main__, sys.exc_info() style. In this case, would this be the correct way to store it in runpy:

try:
    import threading
except ImportError:
    temp_main = None
else:
    local_storage = threading.local()
    local_storage.temp_main = None
    temp_main = local_storage.temp_main

?

Good point about the extra parameter just pushing the problem one layer up the stack rather than completely solving the problem.

However, on further reflection, I've realised that I really don't like having runpy import the threading module automatically, since that means even single-threaded applications run via "-m" will end up initialising the thread support, including the GIL. That's something we try reasonably hard to avoid doing in applications that don't actually need it (it does happen in library modules that genuinely need thread-local storage, such as the decimal module).

If you look at the way Pdb._runscript currently works, it imports __main__ and then cleans it out ready to let the child script run. So replacing that with a simple module level global that refers to the runpy execution namespace would probably be an improvement.

Looking at this use case more closely, though, shows that it isn't as simple as handing the whole task over to the runpy module, as the debugger needs access to the filename before it starts executing code in order to configure the trace function correctly.

That means runpy needs to support a two stage execution process that allows a client script like pdb to retrieve details of the code to be executed, and then subsequently request that it be executed in a specific namespace. My first thought is to switch to a more object-oriented API along the lines of the following:

- get_path_runner()
- get_module_runner()
    These functions would parallel the current run_module() and run_path() functions, but would return a CodeRunner object instead of directly executing the specified module

- CodeRunner.run(module=None)
    This method would actually execute the code, using the specified namespace if given, or an automatic temporary namespace otherwise.

CodeRunner would store sufficient state to support the delayed execution, as well as providing access to key pieces of information (such as the filename) before code execution actually occurs.

pdb could then largely be left alone from a semantic point of view (i.e. still execute everything in the true __main__ module), except that its current code for finding the script to execute would be replaced by a call to runpy.get_runner_for_path(), a new "-m" switch would be added that tweaked that path to invoke runp.get_runner_for_module() instead, the debugger priming step would query the CodeRunner object for the filename, and finally, the actual code execution step would invoke the run() method of the CodeRunner object (passing in __main__ itself as the target module).

Thanks, Nick. Before your last comment, I haven't looked much into Pdb, instead focusing on profile.py and trace.py because they looked like simpler cases. I think the approach with CodeRunner objects would work just fine for profile and trace but Pdb uses run() inherited from Bdb. In order to make it work with a CodeRunner object, it seems run() would have to be reimplemented in Pdb (effectively becoming a 'runCodeRunner()'), and we could probably do without _runscript(). Is that what you had in mind?