classification
Title: uuid.uuid1() is too slow
Type: performance Stage: needs patch
Components: Library (Lib) Versions: Python 3.2, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: alexandre.vassalotti, grooverdan, thijs, wangchun (4)
Priority: low Keywords patch

Created on 2009-04-30 10:13 by wangchun, last changed 2009-09-23 17:44 by thijs.

Files
File name Uploaded Description Edit Remove
uuid_c_module.patch grooverdan, 2009-09-01 09:12
Messages (5)
msg86840 - (view) Author: Wang Chun (wangchun) Date: 2009-04-30 10:13
uuid.uuid1() currently uses two different ways to generate a uuid. If 
the system call "uuid_generate_time" is available, uuid1() uses the 
system call via the ctypes interface, otherwise, it uses pure Python 
code to generate a uuid. The problem is, the C interface 
"uuid_generate_time" is even slower than the Python code. The ctypes 
interface is too slow. According to my test, it took 55 microseconds to 
generate a uuid via ctypes interface but only 45 microseconds via the 
Python code. I also tried to test the performance of the 
"uuid_generate_time" C API itself. It takes C code 12 microseconds. Most 
of the time were wasted on ctypes. I believe we need to drop ctypes and 
write a Python extensions in C for this job.
msg86841 - (view) Author: Wang Chun (wangchun) Date: 2009-04-30 10:42
This is my test on another faster machine.

$ cat test.py
import sys, time, uuid
N = int(sys.argv[1])
t = time.time()
for x in xrange(N):
    uuid.uuid1()
print('%.3f microseconds' % ((time.time() - t) * 1000000.0 / N))
$ cat test.c
#include <stdio.h>
#include <sys/time.h>
#include <uuid/uuid.h>

int main(int argc, char *argv[])
{
	int i, n;
	double t1, t2;
	uuid_t uuid;
	struct timeval t;
	struct timezone tz;
	sscanf(argv[1], "%d", &n);
	gettimeofday(&t, &tz);
	t1 = (double)t.tv_sec + (double)t.tv_usec / 1000000.0;
	for (i = 0; i < n; i++) {
		uuid_generate_time(uuid);
	}
	gettimeofday(&t, &tz);
	t2 = (double)t.tv_sec + (double)t.tv_usec / 1000000.0;
	printf("%.3f microseconds\n", (t2 - t1) * 1000000.0 / n);
	return 0;
}
$ gcc -l uuid -o test test.c
$ python test.py 50000
25.944 microseconds
$ python test.py 200000
25.810 microseconds
$ python test.py 1000000
25.865 microseconds
$ ./test 50000
0.214 microseconds
$ ./test 200000
0.214 microseconds
$ ./test 1000000
0.212 microseconds
$
msg90201 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) Date: 2009-07-06 23:48
Can you provide a patch?
msg92137 - (view) Author: Daniel Black (grooverdan) Date: 2009-09-01 09:12
This is a slightly crude module version. The speedups were only 10%

Python 3.2a0 (py3k:74612M, Sep  1 2009, 18:11:58)                       
[GCC 4.3.2] on linux2  

Using the same test from Wang Chun:
before:
uuid1(1000000)
101.759 microseconds

after:
uuid1(1000000)
91.663 microseconds

The delays are clearly in the _byte array copying as indicated by the
test below:
>>> import sys, time, uuid
>>> def uu(n):
...      t = time.time()
...      for x in range(n):
...         uuid._uuid_generate_time_fast()
...      print('%.3f microseconds' % ((time.time() - t) * 1000000.0 / n))
...
[72265 refs]
>>> uu(1000000)
13.157 microseconds
[72267 refs]

I would expect fixing this for the ctypes version would have a similar
speedup.
msg92138 - (view) Author: Daniel Black (grooverdan) Date: 2009-09-01 09:29
to prove it a bit more - ctype benchmark
>>> import ctypes, ctypes.util
>>> def uu1(n):
...      t = time.time()
...      _buffer = ctypes.create_string_buffer(16)
...      for x in range(n):
...         uuid._uuid_generate_time(_buffer)
...      print('%.3f microseconds' % ((time.time() - t) * 1000000.0 / n))
...
>>> uu1(1000000)
15.819 microseconds
History
Date User Action Args
2009-09-23 17:44:21thijssetnosy: + thijs
2009-09-01 09:29:29grooverdansetmessages: + msg92138
2009-09-01 09:12:43grooverdansetfiles: + uuid_c_module.patch

nosy: + grooverdan
messages: + msg92137

keywords: + patch
2009-07-06 23:48:15alexandre.vassalottisetpriority: low
versions: + Python 3.2, - Python 2.6, Python 2.5, Python 2.4, Python 3.0, Python 3.1
nosy: + alexandre.vassalotti

messages: + msg90201

stage: needs patch
2009-04-30 10:42:54wangchunsetmessages: + msg86841
2009-04-30 10:13:35wangchuncreate