This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Python argparse doesn't work in the presence of my custom module
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: JP Zhang, peter.otten, xtreak
Priority: normal Keywords:

Created on 2019-04-08 14:59 by JP Zhang, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (5)
msg339646 - (view) Author: JP Zhang (JP Zhang) Date: 2019-04-08 14:59
Github repo for reproducing: https://github.com/zjplab/gc-mc-pytorch/tree/bug, test.py. 

In the presence of my custom data_loader, it will error as unrecognized argument. But without importing it(comment it out) everything is just fine.
msg339647 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2019-04-08 15:07
Can you please post the traceback you are getting and how you are running the script from command line? The custom module has third party dependencies like numpy so it will be helpful if you can attach a reproducer without dependencies to see if it's a problem with CPython or something with custom loader.
msg339649 - (view) Author: JP Zhang (JP Zhang) Date: 2019-04-08 15:16
>>>python test.py --num_epochs 200
usage: test.py [-h] [--data_type DATA_TYPE]
test.py: error: unrecognized arguments: --num_epochs 200

I have a created a colab share link. You can check: https://colab.research.google.com/drive/1TUvt4CCv2d43GD1ccmuRNBJlkbUPXN8Z
msg339654 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2019-04-08 15:39
test.py shown in msg339649 accepts only --data_type DATA_TYPE. Running test.py without import and the non-existent flag would show all the options present like the below I am using from the repo you have shared. You might 


./python.exe ../backups/bpo36561.py --non-existent 100
usage: bpo36561.py [-h] [--mode MODE] [--data_type DATA_TYPE] [--model-path MODEL_PATH]
                   [--data-path DATA_PATH] [--data-shuffle DATA_SHUFFLE] [--batch-size BATCH_SIZE]
                   [--num_epochs NUM_EPOCHS] [--val-step VAL_STEP] [--test-epoch TEST_EPOCH]
                   [--start-epoch START_EPOCH] [--neg-cnt NEG_CNT] [--lr LR] [--beta1 BETA1]
                   [--beta2 BETA2] [--dropout DROPOUT] [--n_critic N_CRITIC] [--emb-dim EMB_DIM]
                   [--hidden HIDDEN] [--nb NB] [--train_path TRAIN_PATH] [--val_path VAL_PATH]
                   [--test_path TEST_PATH]
bpo36561.py: error: unrecognized arguments: --non-existent 100

Looking further the problem might be that importing data_loader imports preprocess module and it has the below code. So this argparse code is executed and not the one from tests.py. Python executes all the top level code while importing. Maybe you can have this inside if __name__ == "__main__" so that this is not used while importing,

parser = argparse.ArgumentParser()
# data
parser.add_argument('--data_type', type=str, default="ml_100k")
args = parser.parse_args()
download_dataset(args.data_type)
preprocess(args.data_type)

Can help so that this is not executed.

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    # data
    parser.add_argument('--data_type', type=str, default="ml_100k")
    args = parser.parse_args()
    download_dataset(args.data_type)
    preprocess(args.data_type)

This is reproducible like this where you expect foo to be printed but due to import bar's argument parser is used : 

➜  gc-mc-pytorch git:(master) ✗ cat /tmp/foo.py
import bar
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--foo', required=True)
parser.parse_args()

➜  gc-mc-pytorch git:(master) ✗ cat /tmp/bar.py
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--bar', required=True)
parser.parse_args()

➜  gc-mc-pytorch git:(master) ✗ python3 /tmp/foo.py
usage: foo.py [-h] --bar BAR
foo.py: error: the following arguments are required: --bar


Either way it's not a bug in Python and it's a side effect of top level code being executed during importing the module. I am closing this as not a bug.
msg339655 - (view) Author: Peter Otten (peter.otten) * Date: 2019-04-08 15:45
That's a bug in your code. You create another ArgumentParser in the toplevel code of preprocess.py. When this module is imported directly or indirectly your script will us this parser to parse the command line first. Minimal example:

$ cat preprocess.py
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--foo")
print(parser.parse_args())
$ cat test.py
import argparse
import preprocess

parser = argparse.ArgumentParser()
parser.add_argument("--bar")
print(parser.parse_args())
$ python3 test.py --bar 42
usage: test.py [-h] [--foo FOO]
test.py: error: unrecognized arguments: --bar 42
$ 

Fix: Protect toplevel code in preprocess.py with

if __name__ == "__main__":
    parser = ...
    ...
History
Date User Action Args
2022-04-11 14:59:13adminsetgithub: 80742
2019-04-08 15:45:25peter.ottensetnosy: + peter.otten
messages: + msg339655
2019-04-08 15:39:26xtreaksetstatus: open -> closed
resolution: not a bug
messages: + msg339654

stage: resolved
2019-04-08 15:16:45JP Zhangsetmessages: + msg339649
2019-04-08 15:07:16xtreaksetnosy: + xtreak
messages: + msg339647
2019-04-08 14:59:10JP Zhangcreate