Author jvdias
Recipients jvdias
Date 2009-04-09.06:22:05
SpamBayes Score 2.52576e-13
Marked as misclassified No
Message-id <1239258129.57.0.360132834791.issue5725@psf.upfronthosting.co.za>
In-reply-to
Content
To Build:

$ gcc -fPIC -shared -o psempy.c psempy.so -I/usr/include/python2.6
-L/usr/lib/python2.6 -lpython2.6 && mv psempy.so psem.so
$ dd if=/dev/urandom of=app1_20090407.01.log bs=1000000 count=1
$ python
>>> import sys, os, re, datetime, psem, psem_example
>>> psem_example.compress_log( "app1", "2009", "04", "07", "01", "bzip",
"app1_20090407.01.log");
0

Example psem.so using program that compresses logs 
named *{YEAR}-${MONTH}-${DAY}* in a psem.* based 
parallel for .
On a 32 2Ghz processor SPARC, the time taken to compress
32 1MB files using the psem parallel-for (for 32 CPUs) 
was really of the order of the time taken to compress 1
1MB file - ie. roughly 1/32nd of the time taken to compress
32 files serially . 
The number of processes was made secure and "run-away" safe 
ONLY because direct access was available to the 
   semop(2), semget(2), and semctl(2) system calls.
Please can Python put this API into sys or I will create a
Python add-on module to do so - let me know whether this is
a good idea or not - thank you, Jason.
History
Date User Action Args
2009-04-09 06:22:10jvdiassetrecipients: + jvdias
2009-04-09 06:22:09jvdiassetmessageid: <1239258129.57.0.360132834791.issue5725@psf.upfronthosting.co.za>
2009-04-09 06:22:07jvdiaslinkissue5725 messages
2009-04-09 06:22:06jvdiascreate