classification
Title: argparse cannot parse shell variable arguments in file-given arguments
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.4
process
Status: closed Resolution: works for me
Dependencies: Superseder:
Assigned To: Nosy List: ZhuangZi, bethard, r.david.murray
Priority: normal Keywords:

Created on 2012-09-07 00:04 by ZhuangZi, last changed 2012-09-10 18:36 by ZhuangZi. This issue is now closed.

Messages (6)
msg169958 - (view) Author: Nat Hillard (ZhuangZi) Date: 2012-09-07 00:04
When using the argparse argument fromfile_prefix_chars to obtain command line arguments from a file, it is not possible to make use of bash environment variables within this file. 

Ideally one would be able to `export BAR='/Users/x/Desktop/bar'`, and then give arguments such as:

--foo
$BAR

and have this correctly expanded at read time to --foo '/Users/x/Desktop/bar'.

To my knowledge, this is currently only possible if you give the arguments directly via the command line:
python test.py --foo $BAR

It would be great to be able to use environment variables within files as well!
msg169965 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-09-07 01:38
I'm not sure this is a good idea.  $BAR is supported on the command line because your shell supports it.  To support it in files, argparse would have to (re)implement shell parsing, and while we do have a parser in the stdlib that can do some of this (shlex), actually implementing environment variable substitution correctly would be a non-trivial undertaking.  And what about the csh users? (Assuming there are any left :)
msg169969 - (view) Author: Nat Hillard (ZhuangZi) Date: 2012-09-07 02:55
Indeed these are all valid points, and as a zsh user myself I certainly considered the alternative shell issue. That said, if it were at all possible, through a combination of os.environ / shlex, subprocess, and even `source` if necessary, to offload this interpretation to the user's shell (pass the shell a given string, receive its interpretation first before further processing), I could see a lot of benefit from this. 

Relatedly, I have seem similar requests for environment variable support in configparser, here: http://grokbase.com/t/python/python-list/03ckhw75xr/parsing-environment-variables-in-configparser-files

This particular proposal is interesting in that it creates an intermediary between the file's environment variable syntax and the shell's own. Presumably if you standardized the presentation of environment variables within this file you could use os.environ once you had recognized it. 

That said, I can understand, however, if this is prohibitively costly to implement, and it may well be that this option is ill suited to the task I am putting it to.
msg169979 - (view) Author: Steven Bethard (bethard) * (Python committer) Date: 2012-09-07 07:29
You could try declaring a type converter and using the type= parameter of add_argument. Your type converter could look something like:

    def expanded_path(arg):
        return os.path.expandvars(arg)

Would that work?
msg169988 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-09-07 16:23
I had forgotten all about os.path.expandvars.  Note, however, that that function is very naive:

  >>> os.path.expandvars("'$HOME'")
  "'/home/rdmurray'"

That is, it is doing unconditional substitution, not parsing shell syntax.  It should work well for simple cases, though.

It might be worth throwing up a trial balloon on python-ideas for adding something to shlex that would do a "better" job of environment variable substitution for the version of shell syntax that shlex supports, which would therefore become a sort of platform-independent syntax for doing this.
msg170213 - (view) Author: Nat Hillard (ZhuangZi) Date: 2012-09-10 18:36
Thank you, everyone. Defining a new type for this is just what I needed. No additional modifications are necessary on top of this new type.
History
Date User Action Args
2012-09-10 18:36:33ZhuangZisetstatus: open -> closed
resolution: works for me
messages: + msg170213
2012-09-07 16:23:54r.david.murraysetmessages: + msg169988
2012-09-07 07:29:13bethardsetmessages: + msg169979
2012-09-07 02:55:26ZhuangZisetmessages: + msg169969
title: argparse cannot parse bash variable arguments in file-given arguments -> argparse cannot parse shell variable arguments in file-given arguments
2012-09-07 01:38:18r.david.murraysetversions: + Python 3.4, - Python 2.7
nosy: + r.david.murray

messages: + msg169965

type: behavior -> enhancement
2012-09-07 00:04:51ZhuangZicreate