classification
Title: The unittest module diff is missing/forgetting/not putting newline before + and ? for some inputs
Type: behavior Stage:
Components: Tests, Windows Versions: Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: addons_zz, ezio.melotti, michael.foord, ned.deily, paul.moore, rbcollins, tim.golden, xtreak, zach.ware
Priority: normal Keywords:

Created on 2019-01-08 21:57 by addons_zz, last changed 2019-01-09 11:04 by addons_zz.

Messages (8)
msg333258 - (view) Author: Addons Zz (addons_zz) Date: 2019-01-08 21:57
Create this program and run with `Python 3.6.3`:
```python
import unittest

class StdErrUnitTests(unittest.TestCase):

    def test_function_name(self):
        expected = "testing.main_unit_tests.test_dictionaryBasicLogging:416 - dictionary\n" \
                "testing.main_unit_tests.test_dictionaryBasicLogging:417 - dictionary {1: 'defined_chunk'}"

        actual = "15:49:35:912.348986 - testing.main_unit_tests - dictionary\n" \
                "15:49:35:918.879986 - testing.main_unit_tests - dictionary {1: 'defined_chunk'}"

        self.assertEqual(expected, actual)

if __name__ == '__main__':
    unittest.main()
```

### Actual output

```diff
F
======================================================================
FAIL: test_function_name (__main__.StdErrUnitTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\bug_assert_unittests.py", line 13, in test_function_name
    self.assertEqual(expected, actual)
AssertionError: "testing.main_unit_tests.test_dictionaryBa[114 chars]nk'}" != "15:49:35:912.348986 - testing.main_unit_t[94 chars]nk'}"
- testing.main_unit_tests.test_dictionaryBasicLogging:416 - dictionary
- testing.main_unit_tests.test_dictionaryBasicLogging:417 - dictionary {1: 'defined_chunk'}+ 15:49:35:912.348986 - testing.main_unit_tests - dictionary
+ 15:49:35:918.879986 - testing.main_unit_tests - dictionary {1: 'defined_chunk'}

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)
```

### Expected output

```diff
F
======================================================================
FAIL: test_function_name (__main__.StdErrUnitTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\bug_assert_unittests.py", line 13, in test_function_name
    self.assertEqual(expected, actual)
AssertionError: "testing.main_unit_tests.test_dictionaryBa[114 chars]nk'}" != "15:49:35:912.348986 - testing.main_unit_t[94 chars]nk'}"
- testing.main_unit_tests.test_dictionaryBasicLogging:416 - dictionary
- testing.main_unit_tests.test_dictionaryBasicLogging:417 - dictionary {1: 'defined_chunk'}
+ 15:49:35:912.348986 - testing.main_unit_tests - dictionary
+ 15:49:35:918.879986 - testing.main_unit_tests - dictionary {1: 'defined_chunk'}

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)
```

### Differences between actual and expected output

```diff
@@ -7,7 +7,8 @@
     self.assertEqual(expected, actual)
 AssertionError: "testing.main_unit_tests.test_dictionaryBa[114 chars]nk'}" != "15:49:35:912.348986 - testing.main_unit_t[94 chars]nk'}"
 - testing.main_unit_tests.test_dictionaryBasicLogging:416 - dictionary
-- testing.main_unit_tests.test_dictionaryBasicLogging:417 - dictionary {1: 'defined_chunk'}+ 15:49:35:912.348986 - testing.main_unit_tests - dictionary
+- testing.main_unit_tests.test_dictionaryBasicLogging:417 - dictionary {1: 'defined_chunk'}
++ 15:49:35:912.348986 - testing.main_unit_tests - dictionary
 + 15:49:35:918.879986 - testing.main_unit_tests - dictionary {1: 'defined_chunk'}
```

This kind of bug frequently happens. On this case, it is not putting a new line before the `+`. Other cases, it is not putting a new line before the `?`.
msg333261 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-08 22:38
This doesn't appear to be Windows-specific or related to our test suite, so I updated the tags and added the unittest experts.
msg333265 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-01-08 23:06
Also, if you can, please verify that you can reproduce the problem with a current version of Python 3.  3.7.2 is the current maintenance release and 3.6.8 was the final maintenance release of 3.6.x.
msg333272 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-01-09 01:44
This seems to exist on master. Since they are multilines assertMultiLineEqual is used and is there something incorrect during the diff calculation using ndiff due to absence of '\n'? The current diff is calculated as below with difflib.ndiff and seems to produce incorrect output due to absence of newline. The last change was done for single string comparison with issue9174.

difflib.ndiff calculation done internally for the below reproducer

$ ./python.exe
Python 3.8.0a0 (heads/master:a234e14839, Jan  8 2019, 21:57:35)
[Clang 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import difflib
>>> print(''.join(difflib.ndiff(['a\n', 'b'], ['c\n', 'd'])))
- a
- b+ c
+ d
>>> print(''.join(difflib.ndiff(['a\n', 'b\n'], ['c\n', 'd\n']))) # Possible correct candidate?
- a
- b
+ c
+ d

A simpler reproducer

import unittest

class StdErrUnitTests(unittest.TestCase):

    def test_function_name(self):
        expected = "a\nb"
        actual = "c\nd"

        self.assertEqual(expected, actual)

    def test_function_name_newlines_end(self):
        expected = "a\nb\n"
        actual = "c\nd\n"

        self.assertEqual(expected, actual) # produces extra new line at the diff in the end with \d\n


if __name__ == '__main__':
    unittest.main()

$ ./python.exe ../backups/bpo35687_1.py
FF
======================================================================
FAIL: test_function_name (__main__.StdErrUnitTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "../backups/bpo35687_1.py", line 9, in test_function_name
    self.assertEqual(expected, actual)
AssertionError: 'a\nb' != 'c\nd'
- a
- b+ c
+ d

======================================================================
FAIL: test_function_name_newlines_end (__main__.StdErrUnitTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "../backups/bpo35687_1.py", line 15, in test_function_name_newlines_end
    self.assertEqual(expected, actual)
AssertionError: 'a\nb\n' != 'c\nd\n'
- a
- b
+ c
+ d


----------------------------------------------------------------------
Ran 2 tests in 0.003s

FAILED (failures=2)
msg333273 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-01-09 01:52
Searching further this seems to be reported earlier with issue24780 with caret displayed wrongly which was also due to newline missing. But the sample case reported here is also listed at one of the examples in https://bugs.python.org/issue24780#msg258466 . There is an open patch with review in the issue. I guess this is a duplicate of issue24780 then?

Related sample case at msg258466

======================================================================
FAIL: test_notrailingnewline_2 (__main__.AssertEqualTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 18, in test_notrailingnewline_2
    self.assertEqual("a\nbcdf", "a\nbddg")
AssertionError: 'a\nbcdf' != 'a\nbddg'
  a
- bcdf+ bddg
msg333298 - (view) Author: Addons Zz (addons_zz) Date: 2019-01-09 10:50
The issue is also reproducible, by directly using difflib:
```python
import difflib

def bugged_diff(expected, actual):
    expected = expected.splitlines( 1 )
    actual = actual.splitlines( 1 )

    # diff = difflib.ndiff( expected, actual )
    if expected != actual:
        diff = difflib.context_diff( expected, actual, fromfile='expected input', tofile='actual output', lineterm='\n' )
        return '\n' + ''.join( diff )


if __name__ == '__main__':
    expected = "testing.main_unit_tests.test_dictionaryBasicLogging:416 - dictionary\n" \
            "testing.main_unit_tests.test_dictionaryBasicLogging:417 - dictionary {1: 'defined_chunk'}"

    actual = "15:49:35:912.348986 - testing.main_unit_tests - dictionary\n" \
            "15:49:35:918.879986 - testing.main_unit_tests - dictionary {1: 'defined_chunk'}"

    print( expected, actual )
```

It outputs:
```
testing.main_unit_tests.test_dictionaryBasicLogging:416 - dictionary
testing.main_unit_tests.test_dictionaryBasicLogging:417 - dictionary {1: 'defined_chunk'} 15:49:35:912.348986 - testing.main_unit_tests - dictionary
15:49:35:918.879986 - testing.main_unit_tests - dictionary {1: 'defined_chunk'}
```

But it should be: (still missing the new line on the same place as `assertEqual` does)
```
testing.main_unit_tests.test_dictionaryBasicLogging:416 - dictionary
testing.main_unit_tests.test_dictionaryBasicLogging:417 - dictionary {1: 'defined_chunk'} 
15:49:35:912.348986 - testing.main_unit_tests - dictionary
15:49:35:918.879986 - testing.main_unit_tests - dictionary {1: 'defined_chunk'}
```
msg333299 - (view) Author: Addons Zz (addons_zz) Date: 2019-01-09 10:56
* Correction
```python
import difflib

def bugged_diff(expected, actual):
    expected = expected.splitlines( 1 )
    actual = actual.splitlines( 1 )

    # diff = difflib.ndiff( expected, actual )
    if expected != actual:
        diff = difflib.context_diff( expected, actual, fromfile='expected input', tofile='actual output', lineterm='\n' )
        return '\n' + ''.join( diff )


if __name__ == '__main__':
    expected = "testing.main_unit_tests.test_dictionaryBasicLogging:416 - dictionary\n" \
            "testing.main_unit_tests.test_dictionaryBasicLogging:417 - dictionary {1: 'defined_chunk'}"

    actual = "15:49:35:912.348986 - testing.main_unit_tests - dictionary\n" \
            "15:49:35:918.879986 - testing.main_unit_tests - dictionary {1: 'defined_chunk'}"

    print( bugged_diff( expected, actual ) )
```

Outputs:
```
*** expected input
--- actual output
***************
*** 1,2 ****
! testing.main_unit_tests.test_dictionaryBasicLogging:416 - dictionary
! testing.main_unit_tests.test_dictionaryBasicLogging:417 - dictionary {1: 'defined_chunk'}--- 1,2 ----
! 15:49:35:912.348986 - testing.main_unit_tests - dictionary
! 15:49:35:918.879986 - testing.main_unit_tests - dictionary {1: 'defined_chunk'}
```

May be it should be:
```

*** expected input
--- actual output
***************
*** 1,2 ****
! testing.main_unit_tests.test_dictionaryBasicLogging:416 - dictionary
! testing.main_unit_tests.test_dictionaryBasicLogging:417 - dictionary {1: 'defined_chunk'}
--- 1,2 ----
! 15:49:35:912.348986 - testing.main_unit_tests - dictionary
! 15:49:35:918.879986 - testing.main_unit_tests - dictionary {1: 'defined_chunk'}
```
msg333300 - (view) Author: Addons Zz (addons_zz) Date: 2019-01-09 11:04
I applied the patch from https://bugs.python.org/file44679 and it fixed the issue for this example using the unittest module:
```python
import unittest

class StdErrUnitTests(unittest.TestCase):

    def test_function_name(self):
        expected = "testing.main_unit_tests.test_dictionaryBasicLogging:416 - dictionary\n" \
                "testing.main_unit_tests.test_dictionaryBasicLogging:417 - dictionary {1: 'defined_chunk'}"

        actual = "15:49:35:912.348986 - testing.main_unit_tests - dictionary\n" \
                "15:49:35:918.879986 - testing.main_unit_tests - dictionary {1: 'defined_chunk'}"

        self.assertEqual(expected, actual)

if __name__ == '__main__':
    unittest.main()
```
History
Date User Action Args
2019-01-09 11:04:13addons_zzsetmessages: + msg333300
2019-01-09 10:56:08addons_zzsetmessages: + msg333299
2019-01-09 10:50:33addons_zzsetmessages: + msg333298
2019-01-09 01:52:43xtreaksetmessages: + msg333273
2019-01-09 01:44:42xtreaksetmessages: + msg333272
2019-01-09 00:41:29xtreaksetnosy: + xtreak
2019-01-08 23:06:58ned.deilysetnosy: + ned.deily
messages: + msg333265
2019-01-08 22:39:19steve.dowersetnosy: - steve.dower
2019-01-08 22:38:49steve.dowersetnosy: + ezio.melotti, rbcollins, michael.foord
messages: + msg333261
2019-01-08 21:57:48addons_zzcreate