Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ConfigParser 50x slower in 2.7 #55298

Closed
vlachoudis mannequin opened this issue Feb 1, 2011 · 9 comments
Closed

ConfigParser 50x slower in 2.7 #55298

vlachoudis mannequin opened this issue Feb 1, 2011 · 9 comments
Assignees
@rhettinger
Labels
extension-modules C modules in the Modules dir performance Performance or resource usage

Comments

@vlachoudis
Copy link
Mannequin

vlachoudis mannequin commented Feb 1, 2011

BPO 11089
Nosy @smontanaro, @birkenfeld, @rhettinger, @skrah, @ambv
Files
  • ConfigParserTest.py
  • configparser.patch: Py2.7 patch for ConfigParser.py
  • configparser.patch: configparser.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/rhettinger'
closed_at = <Date 2011-02-21.19:43:02.665>
created_at = <Date 2011-02-01.14:04:42.356>
labels = ['extension-modules', 'performance']
title = 'ConfigParser 50x slower in 2.7'
updated_at = <Date 2011-02-21.19:43:02.664>
user = 'https://bugs.python.org/vlachoudis'

    bugs.python.org fields:

    activity = <Date 2011-02-21.19:43:02.664>
actor = 'rhettinger'
assignee = 'rhettinger'
closed = True
closed_date = <Date 2011-02-21.19:43:02.665>
closer = 'rhettinger'
components = ['Extension Modules']
creation = <Date 2011-02-01.14:04:42.356>
creator = 'vlachoudis'
dependencies = []
files = ['20640', '20643', '20654']
hgrepos = []
issue_num = 11089
keywords = ['patch']
message_count = 9.0
messages = ['127699', '127710', '127713', '127715', '127726', '127761', '127764', '127766', '128986']
nosy_count = 6.0
nosy_names = ['skip.montanaro', 'georg.brandl', 'rhettinger', 'skrah', 'lukasz.langa', 'vlachoudis']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue11089'
versions = ['Python 3.2']
    @vlachoudis
    Copy link
    Mannequin Author

    vlachoudis mannequin commented Feb 1, 2011

    The ConfigParser class in 2.7 is almost >50 times slower than in the 2.6 which for large files it renders it almost unusable. Actually the speed decrease depends on the amount of the stored data

    Results from test program:
    Python 2.7 (r27:82500, Sep 16 2010, 18:02:00)
    on 3.5GHz Fedora14 64bit machine
    ConfigParser 166.307140827
    RawConfigParser 0.1887819767

    Python 2.6.4 (r264:75706, Jun 4 2010, 18:20:31)
    on 3.0GHz Fedora13 64bit machine
    ConfigParser 4.24494099617
    RawConfigParser 0.172905921936

    @vlachoudis vlachoudis mannequin added extension-modules C modules in the Modules dir performance Performance or resource usage labels Feb 1, 2011
    @skrah
    Copy link
    Mannequin

    skrah mannequin commented Feb 1, 2011

    If OrderedDict is used, the test case quickly uses 8GB of memory. With
    this change (I'm not suggesting this as a fix!), the timings are normal:

    Index: Lib/ConfigParser.py
    ===================================================================

    --- Lib/ConfigParser.py (revision 88298)
+++ Lib/ConfigParser.py (working copy)
@@ -92,6 +92,7 @@
 except ImportError:
     # fallback for setup.py which hasn't yet built _collections
     _default_dict = dict
+_default_dict = dict
 
 import re

    @rhettinger
    Copy link
    Contributor

    Commenting-out the c.set(section,"item#%d"%(i),str(i)) calls shows that that is not where the problem lies for the ConfigParser() class.
    The issue seems confined to ConfigParser.get().

    The RawConfigParser() class doesn't seem to have the same issue.

    Looking at the 2.7 code for ConfigParser.get() shows that it is doing a lot more than just getting. For example, it does a full copy of defaults dictionary on every call !?

    I'll look at it more shortly.

    @rhettinger
    Copy link
    Contributor

    Attaching a patch that fixes the algorithmic atrocities by using the Chainmap recipe:

    http://code.activestate.com/recipes/305268-chained-map-lookups

    @rhettinger
    Copy link
    Contributor

    Fixed for 2.7 in r88318. Will make a similar fix for 3.1.4 and for 3.2.1.

    @rhettinger rhettinger assigned rhettinger and unassigned ambv Feb 2, 2011
    @rhettinger
    Copy link
    Contributor

    Attaching patch for Python 3.2.

    Georg, I was think of waiting for 3.2.1 for this one, but it can go into 3.2.0 RC2 if you prefer.

    @rhettinger rhettinger assigned birkenfeld and unassigned rhettinger Feb 2, 2011
    @birkenfeld
    Copy link
    Member

    3.2.1 should be fine.

    @rhettinger
    Copy link
    Contributor

    Fixed 3.1 in r88323.

    @rhettinger rhettinger assigned rhettinger and unassigned birkenfeld Feb 2, 2011
    @rhettinger
    Copy link
    Contributor

    See r88469 and r88470.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @rhettinger rhettinger

    Labels
    extension-modules C modules in the Modules dir performance Performance or resource usage
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @ambv @birkenfeld @rhettinger