Message258275
Eric, Steven,
During further testing I was not able to find any real evidence that the statement I was focused on had a real performance issue.
As I did more testing I noticed that appending data to the file slowed down. The file grew initially with ~30-50KB increments and around 500KB it had slowed down to ~3-5KB/s, until around 1MB the file grew at ~1KB/s. I found this to be odd and because Steven had mentioned other processes, I started looking at some other statements.
After quite a lot of trial and error, I was able to use single quotes and increase my performance to acceptable levels.
Example 3 below is how I resolved it.
Can you explain to me why there was a performance penalty in example 2 ?
Python did something under the hood that I am overlooking.
Did conv.escape_string() change something about columnvalue, so that adding a single quote before and after it introduced some add behavior with writing to file ? I am not an expert on Python and remember reading something about Dynamic typing.
Example 1: Fast performance, variable s is not encapsulated with single quotes
6.5MB parsed in ~1 minute.
for key in listkeys:
keyvalue = self.recordstats[key]
fieldtype = keyvalue[0]
columnvalue = record[key]
columnvalue = conv.escape_string(columnvalue)
if (count > 1):
s = "{0},".format(columnvalue) # No single quotes
else
s = "{0},".format(columnvalue) # No single quotes
count -= 1
Append s to file.
Example 2: Slow performance, pre- and post-fixed variable s with single quotes
6.5MB parsed in 35 minutes.
for key in listkeys:
keyvalue = self.recordstats[key]
fieldtype = keyvalue[0]
columnvalue = record[key]
columnvalue = conv.escape_string(columnvalue)
if (count > 1):
s = "'{0}',".format(columnvalue) # Added single quotes
else
s = "'{0}',".format(columnvalue) # Added single quotes
count -= 1
Append s to file.
Example 3: Fast performance, variable columnvalue is pre- and post-fixed with single quotes
6.5MB parsed in !45 seconds.
for key in listkeys:
keyvalue = self.recordstats[key]
fieldtype = keyvalue[0]
columnvalue = record[key]
columnvalue = conv.escape_string("'" + columnvalue + "'") # Moved single quotes to this statement.
if (count > 1):
s = "{0},".format(columnvalue)
else
s = "{0},".format(columnvalue)
count -= 1
Append s to file. |
|
Date |
User |
Action |
Args |
2016-01-15 07:56:39 | poostenr | set | recipients:
+ poostenr, eric.smith, steven.daprano, ubehera |
2016-01-15 07:56:39 | poostenr | set | messageid: <1452844599.03.0.391742164413.issue26118@psf.upfronthosting.co.za> |
2016-01-15 07:56:39 | poostenr | link | issue26118 messages |
2016-01-15 07:56:38 | poostenr | create | |
|