classification
Title: trace/profile conflict with the use of sys.modules[__name__]
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.6, Python 3.4, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: belopolsky Nosy List: belopolsky, bli, davin, eric.snow, ezio.melotti, flox, georg.brandl, ncoghlan, serhiy.storchaka, tati_alchueyr
Priority: normal Keywords: patch

Created on 2010-09-21 18:17 by belopolsky, last changed 2017-01-30 14:08 by davin.

Files
File name Uploaded Description Edit
issue5758_trace_execute_other_modules_main_v0.patch tati_alchueyr, 2012-07-20 14:00 review
issue24676.py serhiy.storchaka, 2015-07-21 05:57
Messages (6)
msg117090 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-09-21 18:17
The main() method of trace and profile modules attempt to emulate the environment in which traced code runs when invoked directly, but it fails in several respects.   The specific problem which is the subject of this issue is that while __name__ is set to '__main__' in code globals, sys.modules['__main__'] still point to the trace/profile module.

Among other problems, this conflicts, with a popular idiom used in regression test scripts:

   support.run_unittest(__name__)

For example,

$ python -m trace -c -C trace.d Lib/test/test_optparse.py 

----------------------------------------------------------------------
Ran 0 tests in 0.001s

OK

No tests are ran because run_unittests() looks for test case classes in the trace module and finds none.


This is related to #9323, so I am merging in the nosy list.  See also r83393.
msg165928 - (view) Author: Tatiana Al-Chueyr (tati_alchueyr) * Date: 2012-07-20 14:00
Yesterday I've studied this problem with flavio.ribeiro, and we've started "solving" it. The result of our progress is available at:
issue5758_trace_execute_other_modules_main_v0.patch

The problem of our approach is that any code outside the condition "if __name__ == '__main__'" will be run twice, as we used imp.load_source to obtain trace's analyzed code and redefine sys.modules['__main__'].

In order to provide a better solution, we were considering using lazy module import, e.g:
http://code.activestate.com/recipes/473888-lazy-module-imports/

Any thoughts on this?
msg166466 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-07-26 08:09
This is a tricky one. Long term, the right approach is to migrate all the "scripts that run other scripts" over to runpy, but the runpy API needs work before we can do that (see #9325).

For bug fix purposes though, these modules can borrow some of runpy's infrastructure in order to fake the system state correctly. Specifically, the runpy._TempModule and runpy._ModifiedArgv0 context managers. (_TempModule may need a tweak to allow the module to be used to be passed in rather than always being implicitly created)

See runpy._run_module_code for an example of how to use them.
msg247014 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-07-21 05:57
Yet one problem (with pickle) was reported in issue24676.
msg286484 - (view) Author: (bli) Date: 2017-01-30 11:08
Just to report that this bug seems responsible for failures at cProfiling some code using multiprocessing:

http://stackoverflow.com/q/41892297/1878788
http://stackoverflow.com/q/11512499/1878788
msg286491 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2017-01-30 14:08
Though this issue is specifically concerned with runpy APIs and their impact especially in running unittest test scripts, it's worth commenting here for people who need a workaround in the short term:  code such as that shared in http://stackoverflow.com/q/41892297/1878788 can be made to run happily by creating a second script which imports the first and simply runs the test(s) from there.

In the specific case of the 'forkiter.py' from http://stackoverflow.com/q/41892297/1878788, one would create a 'run_my_tests.py' with the contents:

from forkiter import main

if __name__ == "__main__":
    exit(main())





Now this invocation of cProfile runs happily because pickle is able to see the module where all the needed classes/functions were defined:
python3.6 -m cProfile -o forkiter.prof ./run_my_tests.py
History
Date User Action Args
2017-01-30 14:08:10davinsetnosy: + davin
messages: + msg286491
2017-01-30 11:08:16blisetnosy: + bli
messages: + msg286484
2015-07-21 05:57:54serhiy.storchakasetfiles: + issue24676.py
versions: + Python 2.7, Python 3.4, Python 3.5, Python 3.6, - Python 3.2
nosy: + serhiy.storchaka

messages: + msg247014
2015-07-21 05:52:20serhiy.storchakalinkissue24676 superseder
2012-11-13 05:00:47eric.snowsetnosy: + eric.snow
2012-07-26 08:09:03ncoghlansetmessages: + msg166466
2012-07-20 18:29:09brett.cannonsetnosy: + ncoghlan
2012-07-20 14:00:28tati_alchueyrsetfiles: + issue5758_trace_execute_other_modules_main_v0.patch

nosy: + tati_alchueyr
messages: + msg165928

keywords: + patch
2012-07-15 03:53:59eli.benderskysetnosy: - eli.bendersky
2010-09-21 18:17:43belopolskycreate