This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author poostenr
Recipients eric.smith, poostenr, steven.daprano, ubehera
Date 2016-01-15.07:56:38
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1452844599.03.0.391742164413.issue26118@psf.upfronthosting.co.za>
In-reply-to
Content
Eric, Steven,

During further testing I was not able to find any real evidence that the statement I was focused on had a real performance issue.

As I did more testing I noticed that appending data to the file slowed down. The file grew initially with ~30-50KB increments and around 500KB it had slowed down to ~3-5KB/s, until around 1MB the file grew at ~1KB/s. I found this to be odd and because Steven had mentioned other processes, I started looking at some other statements.

After quite a lot of trial and error, I was able to use single quotes and increase my performance to acceptable levels.
Example 3 below is how I resolved it.

Can you explain to me why there was a performance penalty in example 2 ?
Python did something under the hood that I am overlooking.

Did conv.escape_string() change something about columnvalue, so that adding a single quote before and after it introduced some add behavior with writing to file ? I am not an expert on Python and remember reading something about Dynamic typing. 

Example 1: Fast performance, variable s is not encapsulated with single quotes
6.5MB parsed in ~1 minute.
for key in listkeys:
    keyvalue = self.recordstats[key]
    fieldtype   = keyvalue[0]
    columnvalue = record[key]
    columnvalue = conv.escape_string(columnvalue)
    if (count > 1):
        s = "{0},".format(columnvalue)  # No single quotes
    else
        s = "{0},".format(columnvalue)  # No single quotes
    count -= 1
    Append s to file.

Example 2: Slow performance, pre- and post-fixed variable s with single quotes
6.5MB parsed in 35 minutes.
for key in listkeys:
    keyvalue = self.recordstats[key]
    fieldtype   = keyvalue[0]
    columnvalue = record[key]
    columnvalue = conv.escape_string(columnvalue)
    if (count > 1):
        s = "'{0}',".format(columnvalue) # Added single quotes
    else
        s = "'{0}',".format(columnvalue) # Added single quotes
    count -= 1
    Append s to file.

Example 3: Fast performance, variable columnvalue is pre- and post-fixed with single quotes
6.5MB parsed in !45 seconds.
for key in listkeys:
    keyvalue = self.recordstats[key]
    fieldtype   = keyvalue[0]
    columnvalue = record[key]
    columnvalue = conv.escape_string("'" + columnvalue + "'") # Moved single quotes to this statement.
    if (count > 1):
        s = "{0},".format(columnvalue)
    else
        s = "{0},".format(columnvalue)
    count -= 1
    Append s to file.
History
Date User Action Args
2016-01-15 07:56:39poostenrsetrecipients: + poostenr, eric.smith, steven.daprano, ubehera
2016-01-15 07:56:39poostenrsetmessageid: <1452844599.03.0.391742164413.issue26118@psf.upfronthosting.co.za>
2016-01-15 07:56:39poostenrlinkissue26118 messages
2016-01-15 07:56:38poostenrcreate