classification
Title: unittest assertEqual difference output foiled by newlines
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Elena.Oat, adchanw, anish.shah, chris.jerdonek, ezio.melotti, loewis, michael.foord, nanjekyejoannah, pynewbie, r.david.murray, rbcollins, tim.peters, xtreak
Priority: normal Keywords: easy, patch

Created on 2015-08-02 16:49 by chris.jerdonek, last changed 2019-01-14 15:30 by xtreak.

Files
File name Uploaded Description Edit
test.py pynewbie, 2016-01-17 01:22 Test case to reproduce the bug
test2.py Elena.Oat, 2016-01-17 13:42
issue24780.patch anish.shah, 2016-02-02 16:59 review
fix_24780.patch adchanw, 2016-09-15 22:29 review
Pull Requests
URL Status Linked Edit
PR 11548 open nanjekyejoannah, 2019-01-14 06:17
PR 11548 open nanjekyejoannah, 2019-01-14 06:17
PR 11548 open nanjekyejoannah, 2019-01-14 06:18
Messages (14)
msg247883 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2015-08-02 16:49
When newlines are present, the error message displayed by unittest's self.assertEqual() to show where strings differ can be nonsensical.  For example, the caret symbol can show up in a strange location.

The first example below shows a case where things work correctly.  The second shows a newline case with the confusing display.


======================================================================
FAIL: test1
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/chris/***/test.py", line 66, in test1
    self.assertEqual("abc", "abd")
AssertionError: 'abc' != 'abd'
- abc
?   ^
+ abd
?   ^


======================================================================
FAIL: test2
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/chris/***/test.py", line 69, in test2
    self.assertEqual("\nabcx", "\nabdx")
AssertionError: '\nabcx' != '\nabdx'
  
- abcx?   ^
+ abdx?   ^
msg247947 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-08-03 19:44
In this particular case the problem is the lack of a trailing newline on the input string.  It might be possible to improve the extended display algorithm by making sure there is a new line before the carrot line, but care must be taken to account for the cases where one string ends with newline and the other doesn't.  I think this problem only applies to strings that have no trailing newline.
msg258132 - (view) Author: anchal agarwal (pynewbie) Date: 2016-01-13 06:04
There is another case where the error message displayed by self.assertEqual() is weird. 

======================================================================
FAIL: test_newline_1 (__main__.AssertEqualTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 9, in test_newline_1
    self.assertEqual("\n abc", "\n abd")
AssertionError: '\n abc' != '\n abd'
  
-  abc?    ^
+  abd?    ^


======================================================================
FAIL: test_newline_2 (__main__.AssertEqualTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 12, in test_newline_2
    self.assertEqual("\nabc", "\nabd")
AssertionError: '\nabc' != '\nabd'
  
- abc+ abd


----------------------------------------------------------------------

There is a difference in between "\nabc" and "\n abc" and hence the difference between output
msg258466 - (view) Author: Elena Oat (Elena.Oat) * Date: 2016-01-17 13:42
The issue is not related only to the caret. In fact, as seen in the below output, the issue occurs anytime there's a newline character in the beginning or middle of the string to be compared. 

In short, if a newline is present in the string and it's in the beginning or middle, a newline character should be put at the end of the string, too. This will make the output look sensible. If, however, the newline is not present at the end, the output is not really readable (the new line is missing).

As we (me and Manvi B.) understand, the caret appears in the output only when the strings are similar enough, i.e. their similarity ratio is high enough. Otherwise, compare function doesn't show the carets in places of difference. This can also be seen in test case test_trailingnewline_2.

This issue occurs, probably, due to using splitlines method.

FFFFFFFF
======================================================================
FAIL: test_notrailingnewline_0 (__main__.AssertEqualTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 8, in test_notrailingnewline_0
    self.assertEqual("abcDefehiJkl", "abcdefGhijkl")
AssertionError: 'abcDefehiJkl' != 'abcdefGhijkl'
- abcDefehiJkl
?    ^  ^  ^
+ abcdefGhijkl
?    ^  ^  ^


======================================================================
FAIL: test_notrailingnewline_1 (__main__.AssertEqualTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 14, in test_notrailingnewline_1
    self.assertEqual("a\nbcdf", "a\nbddf")
AssertionError: 'a\nbcdf' != 'a\nbddf'
  a
- bcdf?  ^
+ bddf?  ^


======================================================================
FAIL: test_notrailingnewline_2 (__main__.AssertEqualTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 18, in test_notrailingnewline_2
    self.assertEqual("a\nbcdf", "a\nbddg")
AssertionError: 'a\nbcdf' != 'a\nbddg'
  a
- bcdf+ bddg

======================================================================
FAIL: test_starting_and_ending_newline_0 (__main__.AssertEqualTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 12, in test_starting_and_ending_newline_0
    self.assertEqual("\nabcDefehiJkl\n", "\nabcdefGhijkl\n")
AssertionError: '\nabcDefehiJkl\n' != '\nabcdefGhijkl\n'
  
- abcDefehiJkl
?    ^  ^  ^
+ abcdefGhijkl
?    ^  ^  ^


======================================================================
FAIL: test_startingnewline_0 (__main__.AssertEqualTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 10, in test_startingnewline_0
    self.assertEqual("\nabcDefehiJkl", "\nabcdefGhijkl")
AssertionError: '\nabcDefehiJkl' != '\nabcdefGhijkl'
  
- abcDefehiJkl?    ^  ^  ^
+ abcdefGhijkl?    ^  ^  ^


======================================================================
FAIL: test_trailingnewline_0 (__main__.AssertEqualTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 6, in test_trailingnewline_0
    self.assertEqual("abcDefehiJkl\n", "abcdefGhijkl\n")
AssertionError: 'abcDefehiJkl\n' != 'abcdefGhijkl\n'
- abcDefehiJkl
?    ^  ^  ^
+ abcdefGhijkl
?    ^  ^  ^


======================================================================
FAIL: test_trailingnewline_1 (__main__.AssertEqualTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 16, in test_trailingnewline_1
    self.assertEqual("a\nbcdf\n", "a\nbddf\n")
AssertionError: 'a\nbcdf\n' != 'a\nbddf\n'
  a
- bcdf
?  ^
+ bddf
?  ^


======================================================================
FAIL: test_trailingnewline_2 (__main__.AssertEqualTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 20, in test_trailingnewline_2
    self.assertEqual("a\nbcdf\n", "a\nbddg\n")
AssertionError: 'a\nbcdf\n' != 'a\nbddg\n'
  a
- bcdf
+ bddg


----------------------------------------------------------------------
Ran 8 tests in 0.007s

FAILED (failures=8)
msg259375 - (view) Author: Anish Shah (anish.shah) * Date: 2016-02-02 10:39
I would like to work on this..
msg259403 - (view) Author: Anish Shah (anish.shah) * Date: 2016-02-02 16:59
The problem is in `difflib.ndiff` function. When the string does not have a trailing newline, we get an unreadable output.
After applying my patch, the following is the output of test2.py (submitted by Elena.Oat).



FFFFFFFF
======================================================================
FAIL: test_notrailingnewline_0 (__main__.AssertEqualTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 8, in test_notrailingnewline_0
    self.assertEqual("abcDefehiJkl", "abcdefGhijkl")
AssertionError: 'abcDefehiJkl' != 'abcdefGhijkl'
- abcDefehiJkl
?    ^  ^  ^
+ abcdefGhijkl
?    ^  ^  ^


======================================================================
FAIL: test_notrailingnewline_1 (__main__.AssertEqualTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 14, in test_notrailingnewline_1
    self.assertEqual("a\nbcdf", "a\nbddf")
AssertionError: 'a\nbcdf' != 'a\nbddf'
  a
- bcdf
?  ^
+ bddf
?  ^


======================================================================
FAIL: test_notrailingnewline_2 (__main__.AssertEqualTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 18, in test_notrailingnewline_2
    self.assertEqual("a\nbcdf", "a\nbddg")
AssertionError: 'a\nbcdf' != 'a\nbddg'
  a
- bcdf
+ bddg


======================================================================
FAIL: test_starting_and_ending_newline_0 (__main__.AssertEqualTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 12, in test_starting_and_ending_newline_0
    self.assertEqual("\nabcDefehiJkl\n", "\nabcdefGhijkl\n")
AssertionError: '\nabcDefehiJkl\n' != '\nabcdefGhijkl\n'
  
- abcDefehiJkl
?    ^  ^  ^
+ abcdefGhijkl
?    ^  ^  ^


======================================================================
FAIL: test_startingnewline_0 (__main__.AssertEqualTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 10, in test_startingnewline_0
    self.assertEqual("\nabcDefehiJkl", "\nabcdefGhijkl")
AssertionError: '\nabcDefehiJkl' != '\nabcdefGhijkl'
  
- abcDefehiJkl
?    ^  ^  ^
+ abcdefGhijkl
?    ^  ^  ^


======================================================================
FAIL: test_trailingnewline_0 (__main__.AssertEqualTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 6, in test_trailingnewline_0
    self.assertEqual("abcDefehiJkl\n", "abcdefGhijkl\n")
AssertionError: 'abcDefehiJkl\n' != 'abcdefGhijkl\n'
- abcDefehiJkl
?    ^  ^  ^
+ abcdefGhijkl
?    ^  ^  ^


======================================================================
FAIL: test_trailingnewline_1 (__main__.AssertEqualTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 16, in test_trailingnewline_1
    self.assertEqual("a\nbcdf\n", "a\nbddf\n")
AssertionError: 'a\nbcdf\n' != 'a\nbddf\n'
  a
- bcdf
?  ^
+ bddf
?  ^


======================================================================
FAIL: test_trailingnewline_2 (__main__.AssertEqualTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 20, in test_trailingnewline_2
    self.assertEqual("a\nbcdf\n", "a\nbddg\n")
AssertionError: 'a\nbcdf\n' != 'a\nbddg\n'
  a
- bcdf
+ bddg


----------------------------------------------------------------------
Ran 8 tests in 0.004s

FAILED (failures=8)
msg261713 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2016-03-14 02:28
Thanks for the patch; reviewed in rietvald.
msg275942 - (view) Author: Adrian Chan (adchanw) Date: 2016-09-12 03:42
Is this still being worked on? I have a potential fix for this.
msg276633 - (view) Author: Adrian Chan (adchanw) Date: 2016-09-15 22:29
I've attached a potential fix for this issue.

While trying to fix this, I noticed that I coudn't assume that I just need to ensure that each line has a newline. If I always ensure each line in diffline has a newline, then the fourth test in testAssertMultilineEqual (in Lib/unittest/test/test_assertions.py) fails because standardMsg in assertMultiLineEqual in Lib/unittest/case.py is just one line without a newline. To sidestep this problem, I made it so that I only ensure there is a newline for each line if and only if there is more than one line in difflines. However, I'm not sure that I can assume there should be a newline in cases similar to the fourth test (where longMessage is set to true and a 'msg' is passed) in testAssertMultilineEqual but where there is more than one line in standardMsg in assertMultiLineEqual.
msg333277 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-01-09 04:25
Looking at the patch and the relevant function this doesn't seem to be a problem with difflib.ndiff but with unittest's display algorithm. This causes confusion about the issue and I propose changing the subject to reflect this unless difflib maintainers think this is an issue with ndiff.
msg333289 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2019-01-09 06:12
When I first created the issue, the title I chose was about unittest ("unittest assertEqual difference output foiled by newlines"), but someone else changed it for some reason. You're welcome to change it back to something more like the original.
msg333295 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-01-09 10:07
Thanks @chris.jerdonek. I have reverted the title to original report. Since CPython now accepts PR if any one of the original authors can convert their patch to a PR with tests then it will be great.
msg333584 - (view) Author: Joannah Nanjekye (nanjekyejoannah) * Date: 2019-01-14 06:21
I have opened a PR for this.
msg333627 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-01-14 15:30
Sorry, I just stumbled upon issue2142 which is a similar report for unique_diff producing wrong output due to missing trailing newlines and could have been the original reason where the title was changed. But since there is a PR now towards adding a newline I think it's good to fix this on unittest side.
History
Date User Action Args
2019-01-14 15:30:04xtreaksetmessages: + msg333627
2019-01-14 06:21:25nanjekyejoannahsetnosy: + nanjekyejoannah
messages: + msg333584
2019-01-14 06:18:29nanjekyejoannahsetstage: needs patch -> patch review
pull_requests: + pull_request11170
2019-01-14 06:18:15nanjekyejoannahsetstage: needs patch -> needs patch
pull_requests: + pull_request11169
2019-01-14 06:17:56nanjekyejoannahsetstage: needs patch -> needs patch
pull_requests: + pull_request11168
2019-01-09 10:07:44xtreaksetmessages: + msg333295
title: difflib.ndiff produces unreadable output when input missing trailing newline -> unittest assertEqual difference output foiled by newlines
2019-01-09 06:12:31chris.jerdoneksetmessages: + msg333289
2019-01-09 04:25:06xtreaksetmessages: + msg333277
2019-01-09 01:55:07xtreaksetnosy: + xtreak

versions: + Python 3.7, Python 3.8, - Python 3.5, Python 3.6
2016-09-17 18:31:29rhettingersetnosy: + tim.peters, loewis
2016-09-15 22:29:38adchanwsetfiles: + fix_24780.patch

messages: + msg276633
2016-09-15 20:19:57ppperrysetcomponents: - Tests
title: unittest assertEqual difference output foiled by newlines -> difflib.ndiff produces unreadable output when input missing trailing newline
2016-09-12 03:42:37adchanwsetnosy: + adchanw
messages: + msg275942
2016-03-14 02:28:06rbcollinssetmessages: + msg261713
2016-02-02 16:59:20anish.shahsetfiles: + issue24780.patch
keywords: + patch
messages: + msg259403
2016-02-02 10:39:11anish.shahsetnosy: + anish.shah
messages: + msg259375
2016-01-17 13:42:40Elena.Oatsetfiles: + test2.py
nosy: + Elena.Oat
messages: + msg258466

2016-01-17 01:31:21ebarrysetcomponents: + Tests
stage: test needed -> needs patch
2016-01-17 01:22:51pynewbiesetfiles: + test.py
2016-01-13 06:04:19pynewbiesetnosy: + pynewbie
messages: + msg258132
2016-01-04 00:06:50ezio.melottisetkeywords: + easy
nosy: + rbcollins, ezio.melotti, michael.foord
stage: test needed

versions: - Python 3.4
2015-08-03 19:44:13r.david.murraysetnosy: + r.david.murray

messages: + msg247947
versions: + Python 2.7, Python 3.5, Python 3.6
2015-08-02 16:49:29chris.jerdonekcreate