classification
Title: uuid.uuid1() is too slow
Type: performance Stage: resolved
Components: Library (Lib) Versions: Python 3.5
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Rework uuid module: lazy initialization and add a new C extension
View: 11063
Assigned To: Nosy List: alexandre.vassalotti, dstanek, grooverdan, r.david.murray, rosslagerwall, serhiy.storchaka, thijs, vstinner, wangchun
Priority: normal Keywords: patch

Created on 2009-04-30 10:13 by wangchun, last changed 2017-09-28 13:26 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
uuid_c_module.patch grooverdan, 2009-09-01 09:12
issue_5885.patch rosslagerwall, 2011-01-22 08:56 Faster patch
Pull Requests
URL Status Linked Edit
PR 3684 closed python-dev, 2017-09-21 11:20
Messages (12)
msg86840 - (view) Author: Wang Chun (wangchun) Date: 2009-04-30 10:13
uuid.uuid1() currently uses two different ways to generate a uuid. If 
the system call "uuid_generate_time" is available, uuid1() uses the 
system call via the ctypes interface, otherwise, it uses pure Python 
code to generate a uuid. The problem is, the C interface 
"uuid_generate_time" is even slower than the Python code. The ctypes 
interface is too slow. According to my test, it took 55 microseconds to 
generate a uuid via ctypes interface but only 45 microseconds via the 
Python code. I also tried to test the performance of the 
"uuid_generate_time" C API itself. It takes C code 12 microseconds. Most 
of the time were wasted on ctypes. I believe we need to drop ctypes and 
write a Python extensions in C for this job.
msg86841 - (view) Author: Wang Chun (wangchun) Date: 2009-04-30 10:42
This is my test on another faster machine.

$ cat test.py
import sys, time, uuid
N = int(sys.argv[1])
t = time.time()
for x in xrange(N):
    uuid.uuid1()
print('%.3f microseconds' % ((time.time() - t) * 1000000.0 / N))
$ cat test.c
#include <stdio.h>
#include <sys/time.h>
#include <uuid/uuid.h>

int main(int argc, char *argv[])
{
	int i, n;
	double t1, t2;
	uuid_t uuid;
	struct timeval t;
	struct timezone tz;
	sscanf(argv[1], "%d", &n);
	gettimeofday(&t, &tz);
	t1 = (double)t.tv_sec + (double)t.tv_usec / 1000000.0;
	for (i = 0; i < n; i++) {
		uuid_generate_time(uuid);
	}
	gettimeofday(&t, &tz);
	t2 = (double)t.tv_sec + (double)t.tv_usec / 1000000.0;
	printf("%.3f microseconds\n", (t2 - t1) * 1000000.0 / n);
	return 0;
}
$ gcc -l uuid -o test test.c
$ python test.py 50000
25.944 microseconds
$ python test.py 200000
25.810 microseconds
$ python test.py 1000000
25.865 microseconds
$ ./test 50000
0.214 microseconds
$ ./test 200000
0.214 microseconds
$ ./test 1000000
0.212 microseconds
$
msg90201 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2009-07-06 23:48
Can you provide a patch?
msg92137 - (view) Author: Daniel Black (grooverdan) * Date: 2009-09-01 09:12
This is a slightly crude module version. The speedups were only 10%

Python 3.2a0 (py3k:74612M, Sep  1 2009, 18:11:58)                       
[GCC 4.3.2] on linux2  

Using the same test from Wang Chun:
before:
uuid1(1000000)
101.759 microseconds

after:
uuid1(1000000)
91.663 microseconds

The delays are clearly in the _byte array copying as indicated by the
test below:
>>> import sys, time, uuid
>>> def uu(n):
...      t = time.time()
...      for x in range(n):
...         uuid._uuid_generate_time_fast()
...      print('%.3f microseconds' % ((time.time() - t) * 1000000.0 / n))
...
[72265 refs]
>>> uu(1000000)
13.157 microseconds
[72267 refs]

I would expect fixing this for the ctypes version would have a similar
speedup.
msg92138 - (view) Author: Daniel Black (grooverdan) * Date: 2009-09-01 09:29
to prove it a bit more - ctype benchmark
>>> import ctypes, ctypes.util
>>> def uu1(n):
...      t = time.time()
...      _buffer = ctypes.create_string_buffer(16)
...      for x in range(n):
...         uuid._uuid_generate_time(_buffer)
...      print('%.3f microseconds' % ((time.time() - t) * 1000000.0 / n))
...
>>> uu1(1000000)
15.819 microseconds
msg126825 - (view) Author: Ross Lagerwall (rosslagerwall) (Python committer) Date: 2011-01-22 08:56
Attached is a patch based on the original patch, meant to have better performance.

On my PC, this:

import sys, time, uuid

def uu(n):
    t = time.time()
    for x in range(n):
        uuid.uuid1()
    print('%.3f microseconds' % ((time.time() - t) * 1000000.0 / n))

uu(50000)

records a time of 38.5 microseconds unpatched (still using ctypes/libuuid) and a time of 16.5 microseconds afterwards.
uuid4() results in an improvement from 30 microseconds to 9 microseconds. From what I could see, what took the most time was the call to UUID() with a bytes object. That's why this patch passes in the uuid as a long.

It also fixes setup.py to check for the uuid.h header.
msg204893 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-12-01 09:16
Instead hexadecimals in _long_from_uuid_t you can use _PyLong_FromByteArray.

However adding new C implemented module has hight cost. I doubt that the speed up of UUID generation is worth this cost.
msg204964 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2013-12-01 21:22
I agree that there is a maintenance cost associated with C extension modules. However, I would certainly be glad if it allowed us to eliminate uses of ctypes in this module because ctypes is quite unsafe and doesn't work well across platforms (though it is admittedly very convenient).
msg240355 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-04-09 17:26
The original report says the ctypes call is slower than the python code used as a fallback.  Would it not, then, be a performance improvement just to drop the ctypes call, without creating a new C module?  Creating a C module would then be a separate enhancement issue if someone thought the performance improvement was enough to justify the module.  Or maybe it could live in the os module?
msg240358 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-04-09 17:34
See also issue11063 and issue20519.
msg240359 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-04-09 17:38
And issue15206. Python implementation has a drawback.
msg303229 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-28 13:26
Hum, there are many open uuid issues which propose similar changes. I close this issue in favor of bpo-11063. Please continue the discussion there.
History
Date User Action Args
2017-09-28 13:26:05vstinnersetstatus: open -> closed

superseder: Rework uuid module: lazy initialization and add a new C extension

nosy: + vstinner
messages: + msg303229
resolution: duplicate
stage: patch review -> resolved
2017-09-21 11:20:00python-devsetpull_requests: + pull_request3673
2015-04-09 17:38:49serhiy.storchakasetmessages: + msg240359
2015-04-09 17:34:59serhiy.storchakasetmessages: + msg240358
2015-04-09 17:26:41r.david.murraysetnosy: + r.david.murray
messages: + msg240355
2013-12-01 21:22:22alexandre.vassalottisetmessages: + msg204964
2013-12-01 09:16:30serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg204893
2013-11-30 21:58:59alexandre.vassalottisetpriority: low -> normal
stage: needs patch -> patch review
versions: + Python 3.5, - Python 2.7, Python 3.2
2011-01-24 15:40:45dstaneksetnosy: + dstanek
2011-01-22 08:56:18rosslagerwallsetfiles: + issue_5885.patch
nosy: + rosslagerwall
messages: + msg126825

2009-09-23 17:44:21thijssetnosy: + thijs
2009-09-01 09:29:29grooverdansetmessages: + msg92138
2009-09-01 09:12:43grooverdansetfiles: + uuid_c_module.patch

nosy: + grooverdan
messages: + msg92137

keywords: + patch
2009-07-06 23:48:15alexandre.vassalottisetpriority: low
versions: + Python 3.2, - Python 2.6, Python 2.5, Python 2.4, Python 3.0, Python 3.1
nosy: + alexandre.vassalotti

messages: + msg90201

stage: needs patch
2009-04-30 10:42:54wangchunsetmessages: + msg86841
2009-04-30 10:13:35wangchuncreate