classification
Title: No way to recover original argv with python -m
Type: enhancement Stage: resolved
Components: Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, Ben.Darnell, barry, brett.cannon, danielsh, flox, georg.brandl, jwilk, merwok, ncoghlan, pitrou, rhettinger
Priority: normal Keywords:

Created on 2012-03-06 06:53 by Ben.Darnell, last changed 2017-03-20 06:34 by ncoghlan. This issue is now closed.

Messages (13)
msg155002 - (view) Author: Ben Darnell (Ben.Darnell) * Date: 2012-03-06 06:53
I have a script which attempts to re-invoke itself using sys.argv, but it fails when run with "python -m package.module".  The problem is that the handling of -m (via the runpy module) rewrites sys.argv as if it were run as "python package/module.py", but the two command lines are not equivalent:  With -m the current directory is inserted at the head of sys.path, but without -m it's the directory containing module.py.  The net effect is that the initial run of "python -m package.module" works as expected, but when it re-runs itself as "python package/module.py" the imports from module.py are effectively relative instead of absolute.

One possible solution would be to provide an immutable sys.__argv__ (by analogy with sys.__stdout__ and friends).
msg155003 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-03-06 07:09
I agree this would be useful.  It would be even more useful to have an __argv__ that includes all command-line flags given to Python, such as -Wi.
msg155044 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-03-07 01:38
This is closely related to PEP 395, since multiprocessing currently hits the same issue in trying to figure out the correct setting for sys.argv0.

I quite like the sys.__argv__ idea for preserving the *exact* underlying command line (Alex Gaynor was recently asking for something similar to this).

In the meantime, it should be possible to work around the problem by running the affected subprocess invocations (i.e. when __main__.__package__ exists and is not empty) with something like:

    launch_template = """
    import runpy, sys
    sys.argv[:] = {argv}
    sys.path[:] = {path}
    runpy._run_module_as_main({main})
    """
    import sys, subprocess, os.path, __main__
    main_base = os.path.basename(__main.__file__).splitext()[0]
    main_ref = __main__.__package__ + "." + main_base
    launch = launch_template.format(argv=sys.argv, path=sys.path, main=main_ref)
    subprocess.call(launch, shell=True, executable=sys.executable)
   
Note: the above isn't tested code, since it's an approach that only occurred to me recently and I haven't had a chance to fully explore it myself. However, if it works, we could make use of it in 2.7 and 3.2 to fix multiprocessing's current misbehaviour on Windows.
msg155053 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2012-03-07 07:36
Instead of sys.__argv__, may I suggest sys.argv_original or somesuch.
msg155073 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-03-07 11:35
In framing a question for Raymond regarding his preference for avoiding the __argv__ name, I realised I agreed with him. My reasoning is that, when a Python process starts, sys.stdin is sys.__stdin__, sys.stdout is sys.__stdout__ and sys.stderr is sys.__stderr__. The dunder versions capture the original values as created by the interpreter initialisation, not the raw OS level file descriptors.

The new attribute proposed here is different - it's not an immutable copy of the original value of sys.argv, it's a *different* sequence altogether. The analogy with the standard stream initial value capture created by the use of sys.__argv__ would actually be misleading rather than helpful.

For the same reason, Raymond's specific argv_original suggestion doesn't really work for me. Alas, I can think of several other possible colours for that particular bikeshed (such as argv_os, argv_main, argv_raw, argv_platform, argv_executable) without having any particular good way of choosing between them.
msg155102 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-03-07 17:41
I agree.  Maybe I may throw "full_argv" or "executable_argv" (i.e. to be used with exec([sys.executable] + sys.executable_arg)) in the air?
msg179845 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-01-13 02:54
For PEP 432, I'm proposing to expose this as sys._configuration.raw_argv
msg179852 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-01-13 09:41
> For PEP 432, I'm proposing to expose this as sys._configuration.raw_argv

I think sys.raw_argv would be as good. It's not really (not only) a
configuration item.
msg179855 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-01-13 10:16
I'm not seeing a good justification for doing anything more, though:

- needing to access this information is an exceedingly niche use case
- I don't see any way for raw_argv to be useful in a cross-implementation manner.
- the exposure as sys._configuration.raw_argv is a natural consequence of the PEP 432 implementation (it's there to allow an embedding application to override the command line arguments seen by CPython)

Elevating it to part of the language standard doesn't make sense, while exposing it as an implementation feature of the CPython initialisation process seems more reasonable.
msg179856 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-01-13 10:19
> I'm not seeing a good justification for doing anything more, though:
> 
> - needing to access this information is an exceedingly niche use case
> - I don't see any way for raw_argv to be useful in a cross-implementation manner.

I expect it to be useful in the "launch (almost) the same command in the
same way" situation.
Not that it happens often :-)
msg179882 - (view) Author: Daniel Shahaf (danielsh) Date: 2013-01-13 15:42
Antoine Pitrou wrote on Sun, Jan 13, 2013 at 10:19:20 +0000:
> 
> Antoine Pitrou added the comment:
> 
> > I'm not seeing a good justification for doing anything more, though:
> > 
> > - needing to access this information is an exceedingly niche use case
> > - I don't see any way for raw_argv to be useful in a cross-implementation manner.
> 
> I expect it to be useful in the "launch (almost) the same command in the
> same way" situation.
> Not that it happens often :-)

What about the "launch a different command with the same interpreter
flags" use-case?  For example, having 'python -E foo.py --foooptions'
want to execute 'python -E bar.py'.

Perhaps it'll be cleaner to expose in state_argv ['python', '-E']; in
sys.argv ['foo.py', '--foooptions']; and have scripts that want to exec
themselves use an idiomatic os.execv(sys.executable, sys.state_argv + sys.argv).
msg180455 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2013-01-23 10:03
My use case is the reloader or restarter. I've initially fallen foul of this when using the cherrypy reloader (which does an execv by building from sys.executable + sys.argv) but I also have web services running which I'd like to restart remotely by forcing them to execv themselves.

It's trivial to retrieve the original command line from the Windows API, shlex.split it and pass it along to execv so this is hardly a showstopper. But it would be useful to have something equivalent built-in.
msg289874 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-03-20 06:34
A few updates here:

* For the specific case of `python -m`, the original argument has been available as `__main__.__spec__.name` since Python 3.4

* Also since Python 3.4, the `multiprocessing` module has correctly handled the -m switch. For more details, see https://docs.python.org/3/whatsnew/3.4.html#multiprocessing and the linked issues.

* However, there are still some cases where it is potentially useful to have access to the full details of how the host Python runtime was invoked, rather than just the arguments that were left after CPython's runtime processing was completed. I filed issue 29857 as a new RFE specifically suggesting a `sys._raw_argv` attribute addressing that question.
History
Date User Action Args
2017-03-20 06:34:44ncoghlansetstatus: open -> closed
versions: + Python 3.4, - Python 3.3
messages: + msg289874

resolution: fixed
stage: resolved
2017-03-19 11:53:16barrysetnosy: + barry
2017-03-19 11:46:56jwilksetnosy: + jwilk
2013-10-24 14:15:07tim.goldensetnosy: - tim.golden
2013-01-23 10:03:18tim.goldensetnosy: + tim.golden
messages: + msg180455
2013-01-13 15:42:34danielshsetmessages: + msg179882
2013-01-13 10:19:20pitrousetmessages: + msg179856
2013-01-13 10:16:29ncoghlansetmessages: + msg179855
2013-01-13 09:41:15pitrousetmessages: + msg179852
2013-01-13 02:54:29ncoghlansetmessages: + msg179845
2013-01-13 00:04:11danielshsetnosy: + danielsh
2012-03-07 17:41:28georg.brandlsetmessages: + msg155102
2012-03-07 11:35:56ncoghlansetmessages: + msg155073
2012-03-07 07:36:18rhettingersetnosy: + rhettinger
messages: + msg155053
2012-03-07 01:38:33ncoghlansetmessages: + msg155044
2012-03-07 00:46:02Arfreversetnosy: + Arfrever
2012-03-06 14:32:13merwoksetnosy: + brett.cannon, ncoghlan, merwok
2012-03-06 08:47:49floxsetnosy: + flox
2012-03-06 07:09:20georg.brandlsetnosy: + georg.brandl, pitrou

messages: + msg155003
versions: + Python 3.3
2012-03-06 06:53:22Ben.Darnellcreate