Title: Adding a testcase for the bug in find_longest_match
Type: Stage:
Components: Tests Versions: Python 2.6
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: Nosy List: georg.brandl, jimjjewett, paulhankin, rtvd
Priority: normal Keywords: patch

Created on 2007-03-11 15:11 by rtvd, last changed 2007-03-19 02:01 by jimjjewett. This issue is now closed.

File name Uploaded Description Edit
python-bug-54269-testcase.diff rtvd, 2007-03-11 15:11 A testcase for the bug #1528074
Messages (4)
msg52171 - (view) Author: Denys Rtveliashvili (rtvd) Date: 2007-03-11 15:11
The find_longest_match method in the difflib's SequenceMatcher has a bug.

The bug is in turn caused by a problem with creating a b2j mapping which should contain a list of indices for each of the list elements in b. However, when the b2j mapping is being created (this is being done in __chain_b method in the SequenceMatcher) the mapping becomes broken. The cause of this is that for the frequently used elements the list of indices is removed and the element is being enlisted in the populardict mapping.

The test case tries to match two strings like:

abbbbbb.... and ...bbbbbbc

The number of b is equal and the find_longest_match should have returned the proper amount. However, in case the number of "b"s is large enough, the method reports that the length of the longest common substring is 0. It simply can't find it.

A bug was raised some time ago on this matter. It's ID is 1528074.

I tried to fix this bug but as a result, the performance drops by a factor of 5-10. Perhaps someone more familiar with Python can find a good fix. For the time being I send this test case (which is broken until the bug is fixed) and I'm going to send a fix next (but the fix makes the method run quite slowly).

The patch diff attached was made against the trunk version of Python.
msg52172 - (view) Author: Paul Hankin (paulhankin) Date: 2007-03-18 17:46
The test correctly fails, and looks right.
msg52173 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2007-03-18 18:28
Added the testcase to Lib/test/ It will be moved to test_difflib as soon as the bug is fixed.
msg52174 - (view) Author: Jim Jewett (jimjjewett) Date: 2007-03-19 02:01
I think the bug is documentation only.  

According to the comments (but not the docstring) find_longest_match returns the longest _interesting_ match.  A single element appearing too often is likely to cause spurious matches, and is therefore not interesting.

I do agree that this should be noted more prominently, so people don't try things like comparing text strings letter-by-letter (where 1% is far too low a threshhold for a 26-character alphabet).

And yes, the comments on popular are correct -- it ignores elements which constitute *more* than 1%.  

I recommend closing this set of tracker items. If you could submit changes to the documentation (docstrings and/or help files; maybe even the comments), I would recommend applying them.
Date User Action Args
2007-03-11 15:11:23rtvdcreate