classification
Title: Provide CPython command line functionality via runpy module
Type: enhancement Stage:
Components: Versions: Python 3.2, Python 2.7
process
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: ncoghlan Nosy List: eric.snow, fwierzbicki, ncoghlan, pitrou
Priority: normal Keywords:

Created on 2009-09-01 10:56 by ncoghlan, last changed 2011-11-29 06:34 by eric.snow. This issue is now closed.

Messages (14)
msg92140 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2009-09-01 10:56
Currently, the runpy._run_module_as_main() function allows a launch
script to mimic Python's -m switch fairly well.

This RFE suggests making it possible to mimic other command line
behaviour of the CPython interpreter in a documented fashion:

run_module_as_main - document and make public the existing
_run_module_as_main function (exposes -m style functionality)

run_path_as_main - accept a filesystem path and execute it as per the
command line rules for executing named scripts, zipfiles and directories
(exposes normal script execution functionality)

run_file_as_main - accept a file-like object and execute it as per the
command line rules for executing stdin (exposes '-' style functionality)

run_code_as_main - accept a string or code object and execute it
(exposes -c style functionality)

The runpy module would also be updated to expose these via its command
line to ensure they achieve their stated goal of allowing a launch
script to mimic the native interpreter behaviour. Due to the legacy of
implementing -m style execution by default, the module command line will
expose filesystem path execution through a '-f' switch.
msg95275 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2009-11-15 07:18
Descoped idea to just provide runpy.run_path (filesystem path equivalent
of runpy.run_module)
msg95280 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2009-11-15 09:10
2.7: r76286
3.2 merge is still a work in progress
msg95311 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2009-11-15 21:59
The windows buildbots are reporting an odd error in test_runpy.  I can't
see by visual inspection why the test is failing:

http://www.python.org/dev/buildbot/stable/builders/x86%20XP-4%20trunk/builds/2615/steps/test/logs/stdio
msg95312 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-11-15 22:20
The test is probably failing because the pattern is interpreted as a regex:
>>> re.match(r'\t', r'\t') is None
True

There is another problem, which is that test_runpy (or runpy itself) is
now leaking references:

test_runpy
beginning 6 repetitions
123456
......
test_runpy leaked [56, 56, 56] references, sum=168
msg95315 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2009-11-15 22:54
The windows errors are most likely due to the fact that I'm not doubling
the backslashes in the file paths before passing them to
assertRaisesRegex() as the regular expression pattern.

The reference leak is also interesting, since there isn't any new C code
here. Although I do add quite a few new entries to the path importer
cache as part of the run_path tests - that might be enough to explain it.
msg95316 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2009-11-15 23:20
Hmm, definitely not the path importer cache - regrtest already takes
that into account by reverting it to its original state before checking
the reference counts.
msg95321 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2009-11-16 04:46
There's another cache in zipimport that wasn't being cleared when
hunting refleaks. I've updated that on the trunk and will include it in
the forward port to 3k (along with a fix for the Windows path separator
problem).
msg95323 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2009-11-16 06:58
Added to Py3k in r76324 (including the fixes made on 2.x after the
original checkin)
msg95407 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-11-17 23:15
Another probably related buildbot failure (again, lack of escaping the
regex pattern):

======================================================================
ERROR: test_directory_error (test.test_runpy.RunPathTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/runpy.py", line
125, in _get_main_module_details
    return _get_module_details(main_name)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/runpy.py", line
115, in _get_module_details
    raise ImportError("No code object available for %s" % mod_name)
ImportError: No code object available for __main__

During handling of the above exception, another exception occurred:

ImportError: can't find '__main__' module in
'/private/var/folders/hm/hmcPs0tuELuc-92a0F0LIk+++TM/-Tmp-/tmpp8zORb'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/test/test_runpy.py",
line 334, in test_directory_error
    self._check_import_error(script_dir, msg)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/test/test_runpy.py",
line 295, in _check_import_error
    self.assertRaisesRegexp(ImportError, msg, run_path, script_name)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/unittest/case.py",
line 869, in assertRaisesRegexp
    callable_obj(*args, **kwargs)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/unittest/case.py",
line 126, in __exit__
    expected_regexp = re.compile(expected_regexp)
  File "/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/re.py",
line 205, in compile
    return _compile(pattern, flags)
  File "/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/re.py",
line 273, in _compile
    p = sre_compile.compile(pattern, flags)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/sre_compile.py", line
491, in compile
    p = sre_parse.parse(p, flags)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/sre_parse.py",
line 692, in parse
    p = _parse_sub(source, pattern, 0)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/sre_parse.py",
line 315, in _parse_sub
    itemsappend(_parse(source, state))
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/sre_parse.py",
line 522, in _parse
    raise error("multiple repeat")
sre_constants.error: multiple repeat

======================================================================
ERROR: test_zipfile_error (test.test_runpy.RunPathTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/runpy.py", line
125, in _get_main_module_details
    return _get_module_details(main_name)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/runpy.py", line
115, in _get_module_details
    raise ImportError("No code object available for %s" % mod_name)
ImportError: No code object available for __main__

During handling of the above exception, another exception occurred:

ImportError: can't find '__main__' module in
'/private/var/folders/hm/hmcPs0tuELuc-92a0F0LIk+++TM/-Tmp-/tmpOprB3K/test_zip.zip'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/test/test_runpy.py",
line 357, in test_zipfile_error
    self._check_import_error(zip_name, msg)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/test/test_runpy.py",
line 295, in _check_import_error
    self.assertRaisesRegexp(ImportError, msg, run_path, script_name)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/unittest/case.py",
line 869, in assertRaisesRegexp
    callable_obj(*args, **kwargs)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/unittest/case.py",
line 126, in __exit__
    expected_regexp = re.compile(expected_regexp)
  File "/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/re.py",
line 205, in compile
    return _compile(pattern, flags)
  File "/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/re.py",
line 273, in _compile
    p = sre_compile.compile(pattern, flags)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/sre_compile.py", line
491, in compile
    p = sre_parse.parse(p, flags)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/sre_parse.py",
line 692, in parse
    p = _parse_sub(source, pattern, 0)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/sre_parse.py",
line 315, in _parse_sub
    itemsappend(_parse(source, state))
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/sre_parse.py",
line 522, in _parse
    raise error("multiple repeat")
sre_constants.error: multiple repeat

----------------------------------------------------------------------
Ran 16 tests in 32.210s
msg95418 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2009-11-18 10:26
Hmm, I think I need to use a more robust regex escaping approach... I
never knew the OS level temp dir creation could get that creative.
msg95420 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2009-11-18 11:32
I fixed the tests to use the proper escaping function from the re module
instead of directly doubling backslashes, so that error shouldn't happen
any more.

(Was it one of the unstable buildbots that picked this up? I didn't see
anything come through on the checkins list)
msg95427 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-11-18 13:42
> (Was it one of the unstable buildbots that picked this up? I didn't see
> anything come through on the checkins list)

Buildbot failures have stopped being e-mailed long ago it seems.
(ah, you really thought our buildbots were all green in all that
time? :P)
msg95457 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2009-11-18 21:43
Antoine Pitrou wrote:
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
>> (Was it one of the unstable buildbots that picked this up? I didn't see
>> anything come through on the checkins list)
> 
> Buildbot failures have stopped being e-mailed long ago it seems.
> (ah, you really thought our buildbots were all green in all that
> time? :P)

I knew the stable list was a lot shorter than the full list, so I
thought it was remotely possible :)
History
Date User Action Args
2011-11-29 06:34:28eric.snowsetnosy: + eric.snow
2009-11-19 08:17:23cjw296setnosy: - cjw296
2009-11-18 21:47:57r.david.murraysetnosy: - r.david.murray
2009-11-18 21:43:56ncoghlansetmessages: + msg95457
2009-11-18 13:42:09pitrousetmessages: + msg95427
2009-11-18 11:32:48ncoghlansetmessages: + msg95420
2009-11-18 10:26:47ncoghlansetmessages: + msg95418
2009-11-17 23:15:48pitrousetmessages: + msg95407
2009-11-16 06:58:45ncoghlansetstatus: open -> closed
resolution: accepted
messages: + msg95323
2009-11-16 04:46:56ncoghlansetmessages: + msg95321
2009-11-15 23:20:03ncoghlansetmessages: + msg95316
2009-11-15 22:54:39ncoghlansetmessages: + msg95315
2009-11-15 22:20:22pitrousetnosy: + pitrou
messages: + msg95312
2009-11-15 21:59:29r.david.murraysetnosy: + r.david.murray
messages: + msg95311
2009-11-15 09:10:24ncoghlansetmessages: + msg95280
2009-11-15 07:18:51ncoghlansetmessages: + msg95275
2009-09-07 23:44:22fwierzbickisetnosy: + fwierzbicki
2009-09-01 11:23:46ncoghlansetversions: + Python 2.7
2009-09-01 11:08:50cjw296setnosy: + cjw296
2009-09-01 10:56:57ncoghlansetassignee: ncoghlan
versions: + Python 3.2
2009-09-01 10:56:37ncoghlancreate