Message96204
It's not a big problem because we can just shell to zcat but compare
these two:
$ time python ./b.py >/dev/null
real 0m10.977s
user 0m7.128s
sys 0m0.888s
$ time python ./a.py >/dev/null
real 1m19.015s
user 1m18.185s
sys 0m0.072s
$ # Notice that the gzip module (a.py) had the benefit of the files
being in a disk cache by too now...
$ cat a.py
import gzip
import os
apt_cache_dir = "/var/cache/apt/apt-file"
for apt_cache_file in os.listdir(apt_cache_dir):
if not apt_cache_file.endswith(".gz"):
continue
f = gzip.open(os.path.join(apt_cache_dir, apt_cache_file))
for line in f:
print line
$ cat b.py
import os
import subprocess
from cStringIO import StringIO
apt_cache_dir = "/var/cache/apt/apt-file"
for apt_cache_file in os.listdir(apt_cache_dir):
if not apt_cache_file.endswith(".gz"):
continue
p = subprocess.Popen(["zcat", os.path.join(apt_cache_dir,
apt_cache_file)],
stdout = subprocess.PIPE)
f = StringIO(p.communicate()[0])
assert p.returncode == 0
for line in f:
print line
Also tried this one just for "completeness":
$ cat c.py
import gzip
import os
from cStringIO import StringIO
apt_cache_dir = "/var/cache/apt/apt-file"
for apt_cache_file in os.listdir(apt_cache_dir):
if not apt_cache_file.endswith(".gz"):
continue
f = gzip.open(os.path.join(apt_cache_dir, apt_cache_file))
f = StringIO(f.read())
for line in f:
print line
But after it had ran (with some thrashing) for 3 and a half minutes I
killed it. |
|
Date |
User |
Action |
Args |
2009-12-10 16:11:25 | asnakelover | set | recipients:
+ asnakelover |
2009-12-10 16:11:25 | asnakelover | set | messageid: <1260461485.52.0.614015732019.issue7471@psf.upfronthosting.co.za> |
2009-12-10 16:11:22 | asnakelover | link | issue7471 messages |
2009-12-10 16:11:21 | asnakelover | create | |
|