Title: doctest 'fancy diff' formats incorrectly strip trailing whitespace
Type: behavior Stage: test needed
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6, Python 3.5, Python 3.4
Status: open Resolution:
Dependencies: Superseder:
Assigned To: CuriousLearner Nosy List: CuriousLearner, jairotrad, r.david.murray, tim.peters
Priority: normal Keywords: easy, patch

Created on 2015-07-29 01:26 by r.david.murray, last changed 2018-11-22 12:47 by CuriousLearner.

File name Uploaded Description Edit
doctest_fancy_diff_trailing_whitespace.diff r.david.murray, 2015-07-29 01:26 review
issue24746.patch jairotrad, 2015-08-17 00:00 review
Pull Requests
URL Status Linked Edit
PR 10639 open CuriousLearner, 2018-11-21 20:02
PR 10639 open CuriousLearner, 2018-11-21 20:02
Messages (8)
msg247552 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-07-29 01:26
I got a doctest failure that when I wrote the output to a file showed two exactly identical lines reported as being different.  Turning off the fancy diff, I could see trailing whitespace on one of the lines.  It turns out that when a fancy diff is requested, doctest explicitly goes through and strips trailing whitespace from the diff lines returned by difflib.  This seems to me to be obviously incorrect.  There is no clue in the changelog why this was done...this goes back to a massive refactoring of doctest that was done for python 2.4, and the fancy diff was introduced at that point, complete with this strange behavior.

I tried to write a test for this but couldn't get it working in the time I was willing to devote to this (I've switched to NDIFF format, which shows the whitespace error even when the actual whitespace is stripped).  Perhaps testing this via doctest isn't the best idea anyway, since it will be far to easy for the trailing whitespace in the test to get accidentally stripped.  I've attached my work as a diff for reference if someone wants to work on this.
msg248695 - (view) Author: Jairo Trad (jairotrad) * Date: 2015-08-17 00:00
I have patched this as explained by David. Also the tests are working. Another test broke because a missing trailing space. I fixed that too.

This is my first patch :D
msg248701 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-08-17 01:12
Unfortunately I was reminded a few days ago that there is a commit hook that prevents patches containing trailing whitespace from being committed to the repository.  So using doctest to test this isn't going to work.

The alternatives are to write a unit test, or to figure out how to write the doctest such there is no trailing whitespace...which would mean capturing the output of doctest.DoctTestRunner instead of letting it go the console, and then comparing it to an explicitly embedded string.
msg248726 - (view) Author: Jairo Trad (jairotrad) * Date: 2015-08-17 14:01
I can write a unittest for this, where should I write it? a new test file?
msg248727 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-08-17 14:04
I believe you can put it in the test_doctest file, and call it using the appropriate runner from from the test_main function.  I haven't tried it though.
msg330214 - (view) Author: Sanyam Khurana (CuriousLearner) * (Python triager) Date: 2018-11-21 20:05
> Unfortunately I was reminded a few days ago that there is a commit hook that prevents patches containing trailing whitespace from being committed to the repository.

Is this still the case?

I tried to figure out writing the test case, but I can't wrap my head around showing the trailing whitespace without putting it anywhere in the code.

Also, can you have a look at the PR, please?
msg330220 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2018-11-21 22:16
To include trailing whitespace on a line in a doctest, _don't_ use raw strings.  Use a regular string, and include add a (one or more) trailing \x20 instead of a space (for example).  For example:

>>> print("a ")

where there's a space at the end of the output line is no good.  It's visually impossible to tell what's intended, and the commit hook should reject it.  But this works fine (a regular string and an escape code):

>>> print("a ")
msg330240 - (view) Author: Sanyam Khurana (CuriousLearner) * (Python triager) Date: 2018-11-22 10:04
Thanks for your suggestion, Tim.

I've fixed the patch and it is ready for review at
Date User Action Args
2018-11-22 12:47:36CuriousLearnersetassignee: CuriousLearner
2018-11-22 10:04:06CuriousLearnersetmessages: + msg330240
2018-11-21 22:16:38tim.peterssetnosy: + tim.peters
messages: + msg330220
2018-11-21 20:05:30CuriousLearnersetversions: + Python 3.7, Python 3.8
nosy: + CuriousLearner

messages: + msg330214

stage: patch review -> test needed
2018-11-21 20:02:29CuriousLearnersetstage: test needed -> patch review
pull_requests: + pull_request9887
2018-11-21 20:02:26CuriousLearnersetstage: test needed -> test needed
pull_requests: + pull_request9886
2015-08-17 14:04:46r.david.murraysetmessages: + msg248727
2015-08-17 14:01:23jairotradsetmessages: + msg248726
2015-08-17 01:12:36r.david.murraysetmessages: + msg248701
2015-08-17 00:00:23jairotradsetfiles: + issue24746.patch
nosy: + jairotrad
messages: + msg248695

2015-07-29 01:26:52r.david.murraycreate