classification
Title: 2to3 is rather slow
Type: performance Stage:
Components: 2to3 (2.x to 3.x conversion tool) Versions:
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: wolever Nosy List: benjamin.peterson, collinwinter, wolever
Priority: normal Keywords: patch

Created on 2008-03-20 05:30 by wolever, last changed 2008-08-21 22:22 by benjamin.peterson. This issue is now closed.

Files
File name Uploaded Description Edit
fixer_head_node_lookup.diff wolever, 2008-03-20 19:55
fixer_head_node_lookup_2.diff wolever, 2008-03-25 16:25
Messages (10)
msg64152 - (view) Author: David Wolever (wolever) * (Python committer) Date: 2008-03-20 05:30
It takes me 10 seconds to run 2to3 on the example file.
msg64171 - (view) Author: Collin Winter (collinwinter) * (Python committer) Date: 2008-03-20 15:29
Yes, and each new fixer makes it slower. The biggest problem is the
pattern matching system, which tends to take 99% of the running time.
I've talked to several people who are interested in optimizing 2to3 as a
GSoC project, though, so I anticipate some speed ups coming through that.
msg64183 - (view) Author: David Wolever (wolever) * (Python committer) Date: 2008-03-20 17:37
Martin and I talked about this, and I'm going to try loading the first 
node of each tree generated by the PATTERNs into a dictionary, then when 
the tree is walked, only fixers which could potentially match the current 
node are executed.
msg64186 - (view) Author: Collin Winter (collinwinter) * (Python committer) Date: 2008-03-20 17:58
Keep in mind that not all fixers use PATTERN, for example fix_ne.
msg64199 - (view) Author: David Wolever (wolever) * (Python committer) Date: 2008-03-20 19:55
A patch so that, on each node, only fixers who's head node could match 
this node are executed.  It's still hacky and ugly, but a decent POC.
msg64489 - (view) Author: Collin Winter (collinwinter) * (Python committer) Date: 2008-03-25 15:42
With the patch applied, all tests in test_fixers fail.
msg64495 - (view) Author: David Wolever (wolever) * (Python committer) Date: 2008-03-25 16:25
Ah, yea -- in my rush to catch my train I must have neglected to look at 
the _results_ of running test.py... I have updated the patch so the fixers 
will work.
msg64501 - (view) Author: Collin Winter (collinwinter) * (Python committer) Date: 2008-03-25 17:43
The updated version of the patch passes the tests and yields a 19-20%
speed increase in the running time of test_all_fixers and the time taken
for 2to3 to translate itself. I've sent David some comments on the
patch, and once it's cleaned up some, this will be checked in.

I'd like to keep the issue open, since there are clearly additional
speed increases to be made.
msg65707 - (view) Author: David Wolever (wolever) * (Python committer) Date: 2008-04-24 00:14
I've cleaned up and applied the patch in r62470.
As Collin suggested, I'm leaving the ticket open to use in the future.
msg71693 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-08-21 22:22
I think we can safely say this has been fixed especially with regards to
fix_imports. Thanks to Nick Edds! It only takes me about 3 seconds to
process the example file.
History
Date User Action Args
2008-08-21 22:22:40benjamin.petersonsetstatus: open -> closed
nosy: + benjamin.peterson
resolution: fixed
messages: + msg71693
2008-04-24 00:14:44woleversetmessages: + msg65707
2008-03-25 17:43:39collinwintersetmessages: + msg64501
2008-03-25 16:25:35woleversetfiles: + fixer_head_node_lookup_2.diff
messages: + msg64495
2008-03-25 15:42:22collinwintersetmessages: + msg64489
2008-03-20 19:55:03woleversetfiles: + fixer_head_node_lookup.diff
keywords: + patch
messages: + msg64199
2008-03-20 17:58:43collinwintersetmessages: + msg64186
2008-03-20 17:37:53woleversetassignee: collinwinter -> wolever
messages: + msg64183
2008-03-20 15:29:10collinwintersetmessages: + msg64171
2008-03-20 05:30:51wolevercreate