Title: ConfigParser load speedup
Type: performance Stage: resolved
Components: Library (Lib) Versions: Python 3.2
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: BreamoreBoy, aioryi, brett.cannon, brian.curtin, lukasz.langa, michael.foord
Priority: normal Keywords: patch

Created on 2009-10-12 12:43 by aioryi, last changed 2010-07-26 02:31 by brian.curtin. This issue is now closed.

File name Uploaded Description Edit
speedfix_71564.patch aioryi, 2009-10-12 12:43 Patch to speed up loading of multi-line options.
issue7113.diff lukasz.langa, 2010-07-23 23:46 Patch for Python 3.2
issue7113_without_itertools.diff lukasz.langa, 2010-07-25 23:41 Patch for Python 3.2 without itertools dependency
Messages (8)
msg93895 - (view) Author: albert hofkamp (aioryi) Date: 2009-10-12 12:43
Current implementation (r71564) uses "'%s\n%s' % (old_val, new_line)" to
merge multi-line options into one string.
For options with many lines, this wastes a lot of CPU power.

Attached patch against r71564 fixes this problem by first building a
list of lines for each loaded option, and after reading the whole file,
merging them with the already loaded data. In that way, the '\n'.join()
can be performed once.
Patched works against test/ (and Python

We have witnessed a reduction from 4 hours to 3 seconds loading time
with Python 2.6 and an option of 800000 lines.
msg111399 - (view) Author: Łukasz Langa (lukasz.langa) (Python committer) Date: 2010-07-23 23:46
Won't happen for Python 2.x but the idea is good. The original patch won't apply for Py3k so I implemented this approach from scratch for Python 3.2. Patch included.

Some microbenchmarks of mine show:
- a 3-5% slowdown for very small files (we can ignore this since it's still a blink of an eye)
- a 10% speedup in usual cases (files the size of a typical Pylons app configuration or a typical buildout)
- more so in unrealistic ones (many values with > 100 lines each)

Brett, a quick summary of changes:
- the idea to introduce a list of lines instead of gluing '%s\n%s' during file reading was introduced
- this approach needs an additional phase of joining the lists to strings after reading is done.
- I used itertools.chain to add the default section for iteration purposes without having to create a separate list. I hope the additional dependency on itertools is no problem.
- I reformatted a bit the changes by fred.drake in r78233 (he added support for valueless keys)
- some other slight formatting changes apply as well
- in the tests I've added a testcase for a file that uses multi-line values heavily. The file is generated temporarily since it's 127kB in size. Creation takes a fraction of a second so shouldn't be noticeable.
msg111495 - (view) Author: Mark Lawrence (BreamoreBoy) Date: 2010-07-24 18:02
@Łukasz: I patched the unit test file only and ran it and all tests passed, I assume that this behaviour is incorrect.
msg111497 - (view) Author: Łukasz Langa (lukasz.langa) (Python committer) Date: 2010-07-24 18:20
Thanks, Mark. The tests should pass for both the old version and the patched one. 

The change in the code involves performance, not functionality. I didn't want to set assertions that are time based so naturally the new test also runs on the old implementation. The thing is that it runs slower.
msg111500 - (view) Author: Mark Lawrence (BreamoreBoy) Date: 2010-07-24 18:34
Łukasz in that case I think we should get this committed as the patch is small and looks clean.  I assume that this will be backported as 2.7 and 3.1 are likely to be around for some years yet.
msg111569 - (view) Author: Łukasz Langa (lukasz.langa) (Python committer) Date: 2010-07-25 23:04
In msg111399 I remarked the new itertools dependency. It seems it is in fact problematic because when building Python from scratch `` which is used to build C extensions is using configparser. And one of the C-only modules is itertools :)

Attached a new patch that doesn't include itertools dependencies. The only difference with the last one is:

< import itertools
<         all_sections = itertools.chain([self._defaults],
<                                        self._sections.values()) 
>         all_sections = [self._defaults]
>         all_sections.extend(self._sections.values()) 
msg111578 - (view) Author: Łukasz Langa (lukasz.langa) (Python committer) Date: 2010-07-25 23:41
Patch from py3k root. Some minor formatting changes suggested by Ezio Melotti included.
msg111590 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2010-07-26 02:31
Committed to py3k in r83154 and release27-maint in r83155.
Date User Action Args
2010-07-26 02:31:21brian.curtinsetstatus: open -> closed

nosy: + brian.curtin
messages: + msg111590

resolution: fixed
stage: patch review -> resolved
2010-07-25 23:41:33lukasz.langasetfiles: + issue7113_without_itertools.diff

messages: + msg111578
2010-07-25 23:40:25lukasz.langasetfiles: - issue7113_without_itertools.diff
2010-07-25 23:04:18lukasz.langasetfiles: + issue7113_without_itertools.diff
nosy: + michael.foord
messages: + msg111569

2010-07-24 18:34:15BreamoreBoysetmessages: + msg111500
2010-07-24 18:20:25lukasz.langasetmessages: + msg111497
2010-07-24 18:02:44BreamoreBoysetnosy: + BreamoreBoy

messages: + msg111495
stage: patch review
2010-07-23 23:46:07lukasz.langasetfiles: + issue7113.diff
versions: + Python 3.2, - Python 2.6
nosy: + brett.cannon, lukasz.langa

messages: + msg111399
2009-10-12 12:43:03aioryicreate