classification
Title: test_argparse failure in interactive mode
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: Trundle, bethard, eric.araujo, ezio.melotti, michael.foord, python-dev, terry.reedy
Priority: normal Keywords: easy

Created on 2011-04-22 03:26 by terry.reedy, last changed 2012-01-16 09:09 by terry.reedy. This issue is now closed.

Messages (18)
msg134259 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-04-22 03:26
(3.1 not checked because it seems not to have test_argparse)
Python 2.7 or 3.2, WinXP these pass:
C:\Programs\Python27>python -m test.regrtest test_argparse
C:\Programs\Python32>python -m test test_argparse
[1/1] test_argparse
1 test OK.

C:\Programs\Python32>python -m test -v test_argparse
C:\Programs\Python32>python -c "from test import test_argparse as t; t.test_main()"
Ran 1536 tests in 6.188s
OK

but interactively (command window interpreter or IDLE shell, 3.2 or 2.7)
>>> from test import test_argparse as t; t.test_main()

gives two failures related to extra spaces in usage string (I added a couple of newlines for clarity). I have no idea if problem is with argparse, test_argparse, regrtest, unittest, or interactive versus batch mode.

======================================================================
FAIL: test_groups_parents (test.test_argparse.TestParentParsers)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Programs\Python32\lib\test\test_argparse.py", line 2217, in test_groups_parents
    '''.format(self.main_program)))
  File "C:\Programs\Python32\lib\test\test_argparse.py", line 29, in assertEqual
    super(TestCase, self).assertEqual(obj1, obj2)
AssertionError:
'usage: [-h] [-w W] [-x X] [-y Y | -z Z]\n\noptional arguments:\n  -h, --help  s [truncated]... != 
'usage:  [-h] [-w W] [-x X] [-y Y | -z Z]\n\noptional arguments:\n  -h, --help   [truncated]...
- usage: [-h] [-w W] [-x X] [-y Y | -z Z]
+ usage:  [-h] [-w W] [-x X] [-y Y | -z Z]
?        +
  
  optional arguments:
    -h, --help  show this help message and exit
    -y Y
    -z Z
  
  g:
    gd
  
    -w W
    -x X


======================================================================
FAIL: test_parent_help (test.test_argparse.TestParentParsers)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Programs\Python32\lib\test\test_argparse.py", line 2188, in test_parent_help
    '''.format(self.main_program)))
  File "C:\Programs\Python32\lib\test\test_argparse.py", line 29, in assertEqual
    super(TestCase, self).assertEqual(obj1, obj2)
AssertionError:
'usage: [-h] [-b B] [--d D] [--w W] [-y Y] a z\n\npositional arguments:\n  a\n   [truncated]... != 
'usage:  [-h] [-b B] [--d D] [--w W] [-y Y] a z\n\npositional arguments:\n  a\n  [truncated]...
- usage: [-h] [-b B] [--d D] [--w W] [-y Y] a z
+ usage:  [-h] [-b B] [--d D] [--w W] [-y Y] a z
?        +
  
  positional arguments:
    a
    z
  
  optional arguments:
    -h, --help  show this help message and exit
    -b B
    --w W
  
  c:
    --d D
  
  x:
    -y Y


----------------------------------------------------------------------
Ran 1536 tests in 29.438s

FAILED (failures=2)
Traceback (most recent call last):
  File "<pyshell#11>", line 1, in <module>
    from test import test_argparse as t; t.test_main()
  File "C:\Programs\Python32\lib\test\test_argparse.py", line 4423, in test_main
    support.run_unittest(__name__)
  File "C:\Programs\Python32\lib\test\support.py", line 1145, in run_unittest
    _run_suite(suite)
  File "C:\Programs\Python32\lib\test\support.py", line 1128, in _run_suite
    raise TestFailed(err)
test.support.TestFailed: multiple errors occurred
msg134284 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-04-22 23:37
If I put the same line I ran interactively in a file and run it

from test import test_argparse as t; t.test_main()

all tests pass. So it is specifically a problem from the interactive prompt.
msg134286 - (view) Author: Andreas Stührk (Trundle) * Date: 2011-04-22 23:51
That happens because argparse uses `os.basename(sys.argv[0])` (per default) as program name, but `sys.argv[0]` is usually a string of length 0 at interactive sessions. The tests use ``"usage: {} ...".format(program_name)`` (note that there will be two spaces for an empty `program_name`) for the expected output, while argparse only outputs one space in that case.
msg134291 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-04-23 01:34
Thanks for the diagnosis. I am glad it is something simple.
msg137068 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-05-27 15:51
I don’t know if we should go out of our way to support running tests in interactive mode.  Running them from regrtest is the recommended way.
msg137076 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2011-05-27 16:22
Unless Terry wants to contribute a fix I suggest closing this.
msg137085 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-05-27 16:58
Ahem. Interactive mode is an approved method of running Python code, along with batch mode. The core interpreter and stdlib modules should run correctly in both modes. So the entire test suite should pass in both modes too. If the tests are written correctly, failure would indicate a bug in the tested component.

That aside, the doc for test/ does not contain 'recommended' and does not discuss running a single test. What I did is the easiest way on Windows.

In this case, following Andreas' remark, the bug is in the test, not the module, in that it miscalculates the expected output in the corner case of a null program name. At lines 2172 and 2205 in the 3.2.0 version of test_argparse.py, changing

'''usage: {} ...
...
'''.format(self.main_program)

to

prog = self.main_program
...
'''usage: {}{}...
...
'''.format(prog, ' ' if prog else '')

fixes the problem.
msg137087 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-05-27 17:04
> Interactive mode is an approved method of running Python code, along
> with batch mode. The core interpreter and stdlib modules should run
> correctly in both modes. So the entire test suite should pass in both
> modes too.
You are right.

> That aside, the doc for test/ does not contain 'recommended' and does
> not discuss running a single test.
Such guidelines belong more in the devguide than the library doc: http://docs.python.org/devguide/runtests

> What I did is the easiest way on Windows.
I cannot argue with you on that :)

> In this case, following Andreas' remark, the bug is in the test, not
> the module, in that it miscalculates the expected output in the
> corner case of a null program name.
Since it’s an easy patch in the test file, +1.
msg137088 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2011-05-27 17:07
> Interactive mode is an approved method of running Python code, along with batch mode. 

That is not guaranteed for any particular piece of Python code in the standard library. In particular it is not amenable to test automation, so it is certainly not a requirement that tests pass in this mode. (What is the *use case* for running a command line argument parser in interactive mode?)

I'm certainly not opposed to fixing tests so that they do pass when run like this, but I disagree that it is a "bug".
msg137094 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-05-27 18:47
Unless the doc for a module explicitly diclaims interactive mode (as does multiproccessing), it should run interactively as documented. Batch and interactive are not mutually exclusive; python -i runs a file in batch mode and switches to interactive mode. IDLE *always* runs files this way!

Interactive exploration is a recommended way to learn Python. I agree that it would be tedious to explore the usage of argparse, for instance, by typing everything at the interactive prompt. But one could, for instance, write a file that puts fake content into sys.argv, sets up option and arg specs, and parses. After running the file in IDLE (or with python -i), one might interactively modify sys.argv or the specs and reparse to see what changes.

In any case, using a module interactively and running its test interactively are different things. If a test cannot run interactively, it should be marked as 'skip if interactive' just as with all the other skip conditions. (Skip if not self.program_name might do it.) But this is all moot for this issue.
msg137099 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2011-05-27 22:02
> it should run interactively as documented.

Where is it documented that all tests will run from the IDLE prompt?

I have *never* heard this claim before. I have nothing against tests supporting this, but those who want it to happen will have to do the work.
msg137112 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-05-28 09:20
Terry, I think you can apply the patch you proposed in msg137085 and close this issue.
If the recommended structure of test files is not documented, a section in the devguide should be added, but that's another issue.  (FWIW I'm not even sure there's a recommended structure, if that's the case we should figure out one, document it, and use it in all the tests.)
msg137141 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-05-28 17:48
I will when I get setup to do that again.
msg150923 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-01-09 07:16
http://docs.python.org/devguide/runtests.html
"If you don’t have easy access to a command line, you can run the test suite from a Python or IDLE shell:
>>> from test import autotest"

However, argparse is the least of the test suite problems on Windows.
(#9116, #11732, other problems)
msg150948 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-01-09 16:26
> "If you don’t have easy access to a command line, you can run the test
> suite from a Python or IDLE shell:
> >>> from test import autotest"

I discovered that after our discussion in this report and added it to the devguide in c18fd0ee23ed.

BTW I agree with Ezio that the fix you proposed should be committed.
msg150988 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-01-09 23:34
New changeset b950267efd59 by Terry Jan Reedy in branch '3.2':
#11906 Make test_argparse work interactively by removing extra space
http://hg.python.org/cpython/rev/b950267efd59
msg150990 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-01-10 00:20
New changeset ec32e6ec16fc by Terry Jan Reedy in branch '2.7':
#11906 Make test_argparse work interactively by removing extra space
http://hg.python.org/cpython/rev/ec32e6ec16fc
msg151333 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2012-01-16 07:12
This was committed on py3k in 4f8c24830a5c. Terry, can the issue be closed?
History
Date User Action Args
2012-01-16 09:09:36terry.reedysetstatus: open -> closed
2012-01-16 07:12:52ezio.melottisetmessages: + msg151333
2012-01-10 00:22:47terry.reedysetassignee: terry.reedy
resolution: fixed
stage: resolved
2012-01-10 00:20:22python-devsetmessages: + msg150990
2012-01-09 23:34:30python-devsetnosy: + python-dev
messages: + msg150988
2012-01-09 16:26:27eric.araujosetmessages: + msg150948
2012-01-09 07:16:15terry.reedysetmessages: + msg150923
2011-05-28 17:48:56terry.reedysetmessages: + msg137141
2011-05-28 09:20:18ezio.melottisetnosy: + ezio.melotti
messages: + msg137112
2011-05-27 22:02:20michael.foordsetmessages: + msg137099
2011-05-27 18:47:54terry.reedysetmessages: + msg137094
2011-05-27 17:07:49michael.foordsetmessages: + msg137088
2011-05-27 17:04:10eric.araujosetmessages: + msg137087
2011-05-27 16:58:41terry.reedysetkeywords: + easy

messages: + msg137085
2011-05-27 16:22:03michael.foordsetmessages: + msg137076
2011-05-27 15:51:57eric.araujosetmessages: + msg137068
title: Test_argparse failure but only in interactive mode -> test_argparse failure in interactive mode
2011-04-23 01:34:01terry.reedysetmessages: + msg134291
2011-04-22 23:51:12Trundlesetnosy: + Trundle
messages: + msg134286
2011-04-22 23:37:48terry.reedysetnosy: + michael.foord
messages: + msg134284
2011-04-22 17:02:07eric.araujosetnosy: + eric.araujo
2011-04-22 03:26:10terry.reedycreate