Title: 2to3 crashes when converting doctest using reduce()
Type: behavior Stage: patch review
Components: 2to3 (2.x to 3.x conversion tool) Versions: Python 3.2, Python 3.3, Python 2.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Aaron.Meurer, VPeric, aldwinaldwin, benjamin.peterson, catalin.iacob, eric.araujo, petri.lehtinen
Priority: normal Keywords: patch

Created on 2011-07-22 15:53 by VPeric, last changed 2019-08-13 09:29 by aldwinaldwin.

File name Uploaded Description Edit
issue12611_test.patch petri.lehtinen, 2011-08-06 19:18 Test case review
Pull Requests
URL Status Linked Edit
PR 15249 open aldwinaldwin, 2019-08-13 09:25
Messages (7)
msg140892 - (view) Author: Vlada Peric (VPeric) Date: 2011-07-22 15:53
2to3 crashes when run on a doctest which uses reduce(). This happens with both 2.7 and 3.2's 2to3. I have the following code in a compatibility file, but even using that it fails with the same error: 

    from functools import reduce
except ImportError:
    reduce = reduce

This is the traceback produced:

Traceback (most recent call last):
  File "/usr/local/bin/2to3", line 6, in <module>
  File "/usr/local/lib/python2.6/lib2to3/", line 173, in main
  File "/usr/local/lib/python2.6/lib2to3/", line 620, in refactor
    items, write, doctests_only)
  File "/usr/local/lib/python2.6/lib2to3/", line 275, in refactor
    self.refactor_file(dir_or_file, write, doctests_only)
  File "/usr/local/lib/python2.6/lib2to3/", line 661, in refactor_file
    *args, **kwargs)
  File "/usr/local/lib/python2.6/lib2to3/", line 321, in refactor_file
    output = self.refactor_docstring(input, filename)
  File "/usr/local/lib/python2.6/lib2to3/", line 500, in refactor_docstring
    indent, filename))
  File "/usr/local/lib/python2.6/lib2to3/", line 530, in refactor_doctest
    assert clipped == [u"\n"] * (lineno-1), clipped
AssertionError: [u'from functools import reduce\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n']
msg141310 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2011-07-28 18:18
I cannot reproduce the crash on neither 2.7 nor 3.2. Can you provide more details; attach the exact python source file that crashes 2to3 and give the complete crashing 2to3 command that you're running.
msg141320 - (view) Author: Aaron Meurer (Aaron.Meurer) Date: 2011-07-28 22:18
Vladimir will need to confirm how to reproduce this exactly, but here is corresponding SymPy issue:

The problem is with the sympy/ntheory/ file at (linking to the file from our release candidate, as a workaround is likely to be pushed to master soon).

Vladimir, can you confirm that this particular version of the file reproduces the problem?
msg141421 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2011-07-30 04:49
Still cannot reproduce with the linked file. Setting as pending until we hear the details from Vladimir.
msg141582 - (view) Author: Vlada Peric (VPeric) Date: 2011-08-02 22:19
Confirmed with the file Aaron linked to. I'm using "2to3-3.2 -w -n -d sympy/ntheory/". This is what Python says about itself:

Python 3.2 (r32:88445, Jun  8 2011, 16:34:06) 
[GCC 4.5.1 20101208 [gcc-4_5-branch revision 167585]] on linux2
msg141727 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2011-08-06 19:18
Attached a patch that adds a test case reproducing the issue. Hopely someone with more experience with lib2to3 can do the actual fixing...
msg142072 - (view) Author: Catalin Iacob (catalin.iacob) * Date: 2011-08-14 18:01
I looked at this and understood why it's happening. I don't know exactly how to fix it though, so here's what I found out.

When a doctest appears in a docstring at line n in a file, RefactorTool.parse_block will return a tree corresponding to n - 1 newline characters followed by the code in the doctest. That tree is refactored by RefactoringTool.refactor_tree which usually returns n - 1 newline characters followed by the refactored doctest. However, for the reduce fixer, the tree returned by refactor_tree starts with from functools import reduce followed by n - 1 newline characters and then the doctest reduce line. The failing assert happens when stripping those newlines because they are expected to be at the beginning of the output while in reality they're after the import line.

So the problem is a mismatch between the expectations of the doctest machinery (refactoring code that starts with some newlines results in code that starts with the same number of newlines) and the reduce fixer which adds an import, imports are added at the beginning of the file, therefore something appears before the newlines. Other fixers could exhibit the same problem.
Date User Action Args
2019-08-13 09:29:13aldwinaldwinsetnosy: + aldwinaldwin
2019-08-13 09:25:17aldwinaldwinsetstage: needs patch -> patch review
pull_requests: + pull_request14970
2011-08-14 18:01:27catalin.iacobsetnosy: + catalin.iacob
messages: + msg142072
2011-08-06 19:18:52petri.lehtinensetfiles: + issue12611_test.patch
type: behavior
messages: + msg141727

keywords: + patch
2011-08-02 22:19:30VPericsetstatus: pending -> open

messages: + msg141582
2011-07-30 04:49:26petri.lehtinensetstatus: open -> pending

messages: + msg141421
2011-07-28 22:18:17Aaron.Meurersetstatus: pending -> open

messages: + msg141320
2011-07-28 18:18:19petri.lehtinensetstatus: open -> pending
nosy: + petri.lehtinen
messages: + msg141310

2011-07-25 19:14:30Aaron.Meurersetnosy: + Aaron.Meurer
2011-07-22 21:19:56eric.araujosetnosy: + benjamin.peterson, eric.araujo
stage: needs patch

versions: + Python 3.3
2011-07-22 15:53:01VPericcreate