Message258252
My initial observations with my Python script using:
s = "{0},".format(columnvalue) # fast
Processed ~360MB of data from 2:16PM - 2:51PM (35 minutes, ~10MB/min)
One particular file 6.5MB took ~1 minute.
When I changed this line of code to:
s = "'{0}',".format(columnvalue) # ~30x slower (1 min. vs 30 min.)
Same particular file of 6.5MB took ~30 minutes (228KB/min).
My Python environment is:
C:\Data>python -V
Python 3.5.1 :: Anaconda 2.4.1 (32-bit)
I did some more testing with a very simplified piece of code, but is not conclusive. But there is a significant jump when I introduce the single quotes where I see a jump from 0m2.410s to 0m3.875s.
$ python -V
Python 3.5.1
//
// s='test'
// for x in range(10000000):
// y = "{0}".format(s)
$ time python test.py
real 0m2.410s
user 0m2.356s
sys 0m0.048s
// s='test'
// for x in range(10000000):
// y = "'%s'" % (s)
$ time python test2.py
real 0m2.510s
user 0m2.453s
sys 0m0.051s
// s='test'
// for x in range(10000000):
// y = "'{0}'".format(s)
$ time python test3.py
real 0m3.875s
user 0m3.819s
sys 0m0.048s |
|
Date |
User |
Action |
Args |
2016-01-14 23:55:40 | poostenr | set | recipients:
+ poostenr, eric.smith, ubehera |
2016-01-14 23:55:40 | poostenr | set | messageid: <1452815740.73.0.365081021544.issue26118@psf.upfronthosting.co.za> |
2016-01-14 23:55:40 | poostenr | link | issue26118 messages |
2016-01-14 23:55:40 | poostenr | create | |
|