classification
Title: 3.3 : test_argparse.py fails 'make test'
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.3, Python 3.4, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Jason.Vas.Dias, bethard, eric.araujo, ezio.melotti, paul.j3, python-dev, r.david.murray, serhiy.storchaka, zach.ware
Priority: normal Keywords:

Created on 2011-04-29 16:15 by Jason.Vas.Dias, last changed 2020-04-27 04:38 by zach.ware. This issue is now closed.

Files
File name Uploaded Description Edit
issue11955.diff ezio.melotti, 2013-03-01 08:47
Messages (11)
msg134779 - (view) Author: Jason Vas Dias (Jason.Vas.Dias) Date: 2011-04-29 16:15
Hi - I've been experiencing many errors trying to build any version
of Python that will pass its test suite - see issues : #11946 , #11954 -
and now I've been advised to raise bugs about each test failure -
hence this bug. For details of my config and build procedure, please
see : issue #11954 .

So, running the new ./python fails test_argparse :

$ LD_LIBRARY_PATH=`pwd` LD_PRELINK=`pwd`/libpython3.3.so.1.0 \
  ./python /usr/src/cpython/Lib/test/test_argparse.py
...
======================================================================
FAIL: test_failures_many_groups_listargs (__main__.TestFileTypeW)     
----------------------------------------------------------------------
Traceback (most recent call last):                                    
  File "/usr/src/cpython/Lib/test/test_argparse.py", line 216, in wrapper
    test_func(self)                                                      
  File "/usr/src/cpython/Lib/test/test_argparse.py", line 235, in test_failures
    raises(ArgumentParserError, parser.parse_args, args)                       
AssertionError: ArgumentParserError not raised by parse_args                   

======================================================================
FAIL: test_failures_many_groups_sysargs (__main__.TestFileTypeW)      
----------------------------------------------------------------------
Traceback (most recent call last):                                    
  File "/usr/src/cpython/Lib/test/test_argparse.py", line 216, in wrapper
    test_func(self)                                                      
  File "/usr/src/cpython/Lib/test/test_argparse.py", line 235, in test_failures
    raises(ArgumentParserError, parser.parse_args, args)                       
AssertionError: ArgumentParserError not raised by parse_args                   

======================================================================
FAIL: test_failures_no_groups_listargs (__main__.TestFileTypeW)       
----------------------------------------------------------------------
Traceback (most recent call last):                                    
  File "/usr/src/cpython/Lib/test/test_argparse.py", line 216, in wrapper
    test_func(self)                                                      
  File "/usr/src/cpython/Lib/test/test_argparse.py", line 235, in test_failures
    raises(ArgumentParserError, parser.parse_args, args)                       
AssertionError: ArgumentParserError not raised by parse_args                   

======================================================================
FAIL: test_failures_no_groups_sysargs (__main__.TestFileTypeW)        
----------------------------------------------------------------------
Traceback (most recent call last):                                    
  File "/usr/src/cpython/Lib/test/test_argparse.py", line 216, in wrapper
    test_func(self)                                                      
  File "/usr/src/cpython/Lib/test/test_argparse.py", line 235, in test_failures
    raises(ArgumentParserError, parser.parse_args, args)                       
AssertionError: ArgumentParserError not raised by parse_args                   

======================================================================
FAIL: test_failures_one_group_listargs (__main__.TestFileTypeW)       
----------------------------------------------------------------------
Traceback (most recent call last):                                    
  File "/usr/src/cpython/Lib/test/test_argparse.py", line 216, in wrapper
    test_func(self)                                                      
  File "/usr/src/cpython/Lib/test/test_argparse.py", line 235, in test_failures
    raises(ArgumentParserError, parser.parse_args, args)                       
AssertionError: ArgumentParserError not raised by parse_args                   

======================================================================
FAIL: test_failures_one_group_sysargs (__main__.TestFileTypeW)        
----------------------------------------------------------------------
Traceback (most recent call last):                                    
  File "/usr/src/cpython/Lib/test/test_argparse.py", line 216, in wrapper
    test_func(self)                                                      
  File "/usr/src/cpython/Lib/test/test_argparse.py", line 235, in test_failures
    raises(ArgumentParserError, parser.parse_args, args)                       
AssertionError: ArgumentParserError not raised by parse_args                   

----------------------------------------------------------------------
Ran 1608 tests in 5.293s                                              

FAILED (failures=6)
Traceback (most recent call last):
  File "/usr/src/cpython/Lib/test/test_argparse.py", line 4712, in <module>
    test_main()                                                            
  File "/usr/src/cpython/Lib/test/test_argparse.py", line 4704, in test_main
    support.run_unittest(__name__)                                          
  File "/usr/src/cpython/Lib/test/support.py", line 1208, in run_unittest   
    _run_suite(suite)                                                       
  File "/usr/src/cpython/Lib/test/support.py", line 1191, in _run_suite     
    raise TestFailed(err)                                                   
test.support.TestFailed: multiple errors occurred 


Now, as someone with not too recent python programming experience,
I find it very difficult to understand exactly what is being
tested by :

class TestFileTypeW(TempDirMixin, ParserTestCase):
    """Test the FileType option/argument type for writing files"""

    def setUp(self):
        super(TestFileTypeW, self).setUp()
        self.create_readonly_file('readonly')

    argument_signatures = [
        Sig('-x', type=argparse.FileType('w')),
        Sig('spam', type=argparse.FileType('w')),
    ]
    failures = ['-x', '', 'readonly']
    successes = [
        ('foo', NS(x=None, spam=WFile('foo'))),
        ('-x foo bar', NS(x=WFile('foo'), spam=WFile('bar'))),
        ('bar -x foo', NS(x=WFile('foo'), spam=WFile('bar'))),
        ('-x - -', NS(x=sys.stdout, spam=sys.stdout)),
    ]


But it seems at least one bug is self-evident : test_argparse.py's
error messages are essentially meaningless, and contain no useful
information (beyond the name) about what failed or why - they are
all essentially duplicates of :

Traceback (most recent call last):
  File "/usr/src/cpython/Lib/test/test_argparse.py", line 216, in wrapper
    test_func(self)
  File "/usr/src/cpython/Lib/test/test_argparse.py", line 235, in test_failures
    raises(ArgumentParserError, parser.parse_args, args)

So why emit 6 copies of the same meaningless message ? 

I'm curious: how do you expect those error messages to help people
track down the source of the bug when every error message contains
the same data and line numbers, and they are line numbers not
of specific tests, but of some "error handler" routine ?
msg134792 - (view) Author: Jason Vas Dias (Jason.Vas.Dias) Date: 2011-04-29 17:15
Aha ! the test succeeds as a non root (super-) user .
This is because as a root user I can override w bit
settings on directories I own: see issue #11956 
for fix I applied to test_import.py to fix same
issue. Thanks !
msg134793 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-04-29 17:18
The six error messages tell you that six different tests failed.  Yes, the failures are probably all due to the same cause, but that's just how unit testing works.  (And yes, the argparse tests are a bit more terse and difficult to understand than many of our tests, but that is because they are trying to economically test a lot of different edge cases.)
msg134872 - (view) Author: √Čric Araujo (eric.araujo) * (Python committer) Date: 2011-04-30 15:53
The helpers in test_argparse could be enhanced to print the tested string and expected result in case of failure.  Or the real line number.
msg183245 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-03-01 08:47
The attached patch adds the list of args to the output of assertRaises in case of error, e.g.:
FAIL: test_failures_one_group_sysargs (test.test_argparse.TestPositionalsNargsZeroOrMoreNone)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/wolf/dev/py/3.3/Lib/test/test_argparse.py", line 216, in wrapper
    test_func(self)
  File "/home/wolf/dev/py/3.3/Lib/test/test_argparse.py", line 236, in test_failures
    parser.parse_args(args)
AssertionError: ArgumentParserError not raised : ['foo', 'bar']
msg193970 - (view) Author: paul j3 (paul.j3) * (Python triager) Date: 2013-07-31 01:27
The test names are:

FAIL: test_failures_many_groups_listargs (__main__.TestFileTypeW)     
FAIL: test_failures_many_groups_sysargs (__main__.TestFileTypeW)      
FAIL: test_failures_no_groups_listargs (__main__.TestFileTypeW)       
FAIL: test_failures_no_groups_sysargs (__main__.TestFileTypeW)        
FAIL: test_failures_one_group_listargs (__main__.TestFileTypeW)       
FAIL: test_failures_one_group_sysargs (__main__.TestFileTypeW)    

So they differ by [many_groups, no_groups, one_group] and [listargs, sysargs]

There are about 170 tests in test_argparse that get multiplied by 6 in this way.  If this replication was removed it would cut the number of tests and time to nearly a third.

This replication is not useful.  listargs and sysargs differ only in how a list of arguments is passed to parse_args, either directly, or via sys.argv.  It should be sufficient to test these alternatives once, not 170 times.

'one group' creates an argument group, and adds all arguments to that.  'many groups' creates an argument group for each added argument.  But argument groups are not used for 'parse_args'.  They are only used when formatting help.  By default all the arguments of parser are already put into one of two groups, 'optional arguments' or 'positional arguments'.

There are tests for the help formatting, but they don't use this ParserTesterMetaclass.  

I would recommend removing this test replication.  It is both unnecessary and a source of confusion.  The simplest would be to change the end of the ParserTesterMetaclass.  

    class ParserTesterMetaclass
        def __init__()
        ....
            for add_arguments in [no_groups]:
                for parse_args in [listargs]:
                    AddTests(cls, add_arguments, parse_args)

I suspect there is some confusion among users and developers as to what argument groups do.  

As for the TestFileTypeW error, I suspect it has to do with file writing privileges.  I haven't encountered it, but I running tests in my own directory.
msg224384 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-07-31 08:20
@Ezio will you be following this up, specifically msg193970 ?
msg224785 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-08-04 23:24
New changeset 9a410ae785ff by Ezio Melotti in branch '3.4':
#11955: show the list of args in case of error in test_argparse.
http://hg.python.org/cpython/rev/9a410ae785ff

New changeset 25e634756f79 by Ezio Melotti in branch 'default':
#11955: merge with 3.4.
http://hg.python.org/cpython/rev/25e634756f79
msg224786 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2014-08-04 23:26
I applied my patch, but that doesn't fix the original issue -- it just provides a better error message in case of failure.
msg331689 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-12-12 11:00
I can not reproduce failures. Are they still reproducible?
msg367391 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2020-04-27 04:38
As we've not seen any issues with this on the buildbots, I'm going ahead and closing it.
History
Date User Action Args
2020-04-27 04:38:54zach.waresetstatus: pending -> closed

nosy: + zach.ware
messages: + msg367391

resolution: fixed
stage: needs patch -> resolved
2018-12-12 16:11:53serhiy.storchakasetstatus: open -> pending
2018-12-12 16:10:22BreamoreBoysetstatus: pending -> open
nosy: - BreamoreBoy
2018-12-12 11:00:16serhiy.storchakasetstatus: open -> pending
nosy: + serhiy.storchaka
messages: + msg331689

2014-08-04 23:26:11ezio.melottisetkeywords: - patch

stage: patch review -> needs patch
messages: + msg224786
versions: + Python 3.5
2014-08-04 23:24:52python-devsetnosy: + python-dev
messages: + msg224785
2014-07-31 08:20:41BreamoreBoysetnosy: + BreamoreBoy
messages: + msg224384
2013-07-31 01:27:40paul.j3setnosy: + paul.j3
messages: + msg193970
2013-03-01 08:47:20ezio.melottisetfiles: + issue11955.diff

versions: + Python 3.4
keywords: + patch
nosy: + ezio.melotti

messages: + msg183245
stage: patch review
2011-04-30 15:53:14eric.araujosetnosy: + eric.araujo, bethard
messages: + msg134872
2011-04-29 17:18:48r.david.murraysetnosy: + r.david.murray
messages: + msg134793
2011-04-29 17:15:58Jason.Vas.Diassetmessages: + msg134792
2011-04-29 17:12:58brian.curtinsettype: crash -> behavior
components: + Tests, - Build
2011-04-29 16:15:40Jason.Vas.Diascreate