classification
Title: Replace uuid ctypes usage with an extension module.
Type: enhancement Stage: resolved
Components: ctypes Versions: Python 3.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Rework uuid module: lazy initialization and add a new C extension
View: 11063
Assigned To: Nosy List: amaury.forgeotdarc, belopolsky, gustavo, haypo, meador.inge, ned.deily, pitrou, r.david.murray, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2014-02-05 11:52 by gustavo, last changed 2017-09-28 21:03 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
gc.py gustavo, 2014-02-05 11:52 test program
uuid-no-ctypes.diff gustavo, 2014-02-05 16:16 review
issue20519_10941v2.diff gustavo, 2015-10-26 19:14 review
uuid.diff gustavo, 2015-10-28 13:36 review
Pull Requests
URL Status Linked Edit
PR 3796 merged pitrou, 2017-09-28 10:37
Messages (17)
msg210306 - (view) Author: Gustavo J. A. M. Carneiro (gustavo) * Date: 2014-02-05 11:52
If you try the attached program, you will find that for every iteration the uuid.uuid4() call generates objects that contain reference cycles and need the help of the garbage collector.  This is not nice.  If I make the ctypes module not able to import, then no garbage is generated.

This problem appears in 2.7, 3.3, and 3.4, at least.
msg210316 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-02-05 14:10
I'm not sure that this is a real problem, but have you identified what the cycle is?
msg210320 - (view) Author: Gustavo J. A. M. Carneiro (gustavo) * Date: 2014-02-05 14:28
Well, this isn't a big problem, but I have an application that needs to run with the GC disabled, since it causes pauses of a couple of seconds each time a full collection runs (we have a few million objects allocated).  I will run the GC only once every 3 hours.  So it would be nice if this uuid call didn't generate cycles.

No, I didn't identify the cycle.  All I know is that the garbage below is produced.  If the ctypes module is unavailable, uuid still works but doesn't generate the garbage.

>>> gc.garbage
[(<type '_ctypes.Array'>,), {'raw': <attribute 'raw' of 'c_char_Array_16' objects>, '__module__': 'ctypes', '__dict__': <attribute '__dict__' of 'c_char_Array_16' objects>, '__weakref__': <attribute '__weakref__' of 'c_char_Array_16' objects>, '_length_': 16, '_type_': <class 'ctypes.c_char'>, '__doc__': None, 'value': <attribute 'value' of 'c_char_Array_16' objects>}, <class 'ctypes.c_char_Array_16'>, <attribute '__dict__' of 'c_char_Array_16' objects>, <attribute '__weakref__' of 'c_char_Array_16' objects>, (<class 'ctypes.c_char_Array_16'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_16' objects>, <attribute 'value' of 'c_char_Array_16' objects>, (<type '_ctypes.Array'>,), {'raw': <attribute 'raw' of 'c_char_Array_16' objects>, '__module__': 'ctypes', '__dict__': <attribute '__dict__' of 'c_char_Array_16' objects>, '__weakref__': <attribute '__weakref__' of 'c_char_Array_16' objects>, '_length_': 16, '_type_': <class 'ctypes.c_char'>, '__doc__': None, 'value': <attribute 'value' of 'c_char_Array_16' objects>}, <class 'ctypes.c_char_Array_16'>, <attribute '__dict__' of 'c_char_Array_16' objects>, <attribute '__weakref__' of 'c_char_Array_16' objects>, (<class 'ctypes.c_char_Array_16'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_16' objects>, <attribute 'value' of 'c_char_Array_16' objects>, (<type '_ctypes.Array'>,), {'raw': <attribute 'raw' of 'c_char_Array_16' objects>, '__module__': 'ctypes', '__dict__': <attribute '__dict__' of 'c_char_Array_16' objects>, '__weakref__': <attribute '__weakref__' of 'c_char_Array_16' objects>, '_length_': 16, '_type_': <class 'ctypes.c_char'>, '__doc__': None, 'value': <attribute 'value' of 'c_char_Array_16' objects>}, <class 'ctypes.c_char_Array_16'>, <attribute '__dict__' of 'c_char_Array_16' objects>, <attribute '__weakref__' of 'c_char_Array_16' objects>, (<class 'ctypes.c_char_Array_16'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_16' objects>, <attribute 'value' of 'c_char_Array_16' objects>, (<type '_ctypes.Array'>,), {'raw': <attribute 'raw' of 'c_char_Array_16' objects>, '__module__': 'ctypes', '__dict__': <attribute '__dict__' of 'c_char_Array_16' objects>, '__weakref__': <attribute '__weakref__' of 'c_char_Array_16' objects>, '_length_': 16, '_type_': <class 'ctypes.c_char'>, '__doc__': None, 'value': <attribute 'value' of 'c_char_Array_16' objects>}, <class 'ctypes.c_char_Array_16'>, <attribute '__dict__' of 'c_char_Array_16' objects>, <attribute '__weakref__' of 'c_char_Array_16' objects>, (<class 'ctypes.c_char_Array_16'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_16' objects>, <attribute 'value' of 'c_char_Array_16' objects>]
msg210322 - (view) Author: Gustavo J. A. M. Carneiro (gustavo) * Date: 2014-02-05 14:58
I have narrowed it down to one line of code:
ctypes.create_string_buffer(16)

That is enough to create 7 objects that have reference cycles.

[<class 'ctypes.c_char_Array_16'>, {'__module__': 'ctypes', '__doc__': None, '__weakref__': <attribute '__weakref__' of 'c_char_Array_16' objects>, 'raw': <attribute 'raw' of 'c_char_Array_16' objects>, '_length_': 16, '_type_': <class 'ctypes.c_char'>, 'value': <attribute 'value' of 'c_char_Array_16' objects>, '__dict__': <attribute '__dict__' of 'c_char_Array_16' objects>}, (<class 'ctypes.c_char_Array_16'>, <class '_ctypes.Array'>, <class '_ctypes._CData'>, <class 'object'>), <attribute '__weakref__' of 'c_char_Array_16' objects>, <attribute 'raw' of 'c_char_Array_16' objects>, <attribute 'value' of 'c_char_Array_16' objects>, <attribute '__dict__' of 'c_char_Array_16' objects>]

So maybe the bug is in ctypes itself, not the uuid module.
msg210328 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-02-05 15:44
Yes, I was pretty sure it was in cytpes, from looking at the UUID code.
msg210331 - (view) Author: Gustavo J. A. M. Carneiro (gustavo) * Date: 2014-02-05 16:16
Regardless, if you don't mind, take this patch for Python 3.5 to avoid ctypes, at least in the Linux case (I don't have Windows to test).  Creating a proper extension module is safer and really not that hard...
msg210335 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-02-05 16:57
Thanks.  This looks like a good idea, but I'll leave it to someone with more experience with this module to review it.

Let's change this issue to an enhancement request for uuid instead.  If someone wants to work on the cycle-in-ctypes problem they can open a new issue.
msg210346 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-02-05 20:08
See also issue5885.
msg251509 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-09-24 11:38
In the Python stdlib, we try to avoid ctypes to use instead a C extension module. So I like the idea of a new optional _uuid module.

I reviewed the attached patch.
msg251510 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-09-24 11:38
@Gustavo: Can you please take my remarks in account and rebase your change on the default branch?
msg253493 - (view) Author: Gustavo J. A. M. Carneiro (gustavo) * Date: 2015-10-26 19:14
This patch fixes the Mac OS X issue @haypo pointed out.
msg253494 - (view) Author: Gustavo J. A. M. Carneiro (gustavo) * Date: 2015-10-26 19:25
One issue of note is regarding generate_time().  Originally I found ctypes bindings for this function, so I wrapped it as well in the extension module.  However, it doesn't appear to be used...
msg253511 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-10-27 02:35
New review (question for Windows).
msg253598 - (view) Author: Gustavo J. A. M. Carneiro (gustavo) * Date: 2015-10-28 13:36
New patch that:

1. adds assert(sizeof(uuid_t) == 16); to the extension module;

2. fixes the code path when ctypes has to be used instead of the extension module (needed a bit more refactoring, apologies if it makes the diff harder to read);

3. Adjusts the uuid module tests to account for the possibility of ctypes not being imported.
msg303219 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-09-28 11:18
https://github.com/python/cpython/pull/3796 updates the patch for 3.7 and improves on it a bit by making initialization lazy.
msg303233 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-09-28 13:29
There are too many uuid open issues proposing similar changes. I mark this issue as a duplicate of bpo-11063 to avoid splitted discussions. Please continue the discussion there!
msg303280 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-09-28 21:03
New changeset a106aec2ed6ba171838ca7e6ba43c4e722bbecd1 by Antoine Pitrou in branch 'master':
bpo-11063, bpo-20519: avoid ctypes and improve import time for uuid (#3796)
https://github.com/python/cpython/commit/a106aec2ed6ba171838ca7e6ba43c4e722bbecd1
History
Date User Action Args
2017-09-28 21:03:08pitrousetmessages: + msg303280
2017-09-28 13:29:10hayposetstatus: open -> closed
superseder: Rework uuid module: lazy initialization and add a new C extension
messages: + msg303233

resolution: duplicate
stage: patch review -> resolved
2017-09-28 11:18:33pitrousetnosy: + pitrou
messages: + msg303219
2017-09-28 11:17:07pitrousetversions: + Python 3.7, - Python 3.5
2017-09-28 10:37:57pitrousetpull_requests: + pull_request3781
2015-10-28 13:36:07gustavosetfiles: + uuid.diff

messages: + msg253598
2015-10-27 02:35:24hayposetmessages: + msg253511
2015-10-26 19:25:11gustavosetmessages: + msg253494
2015-10-26 19:14:38gustavosetfiles: + issue20519_10941v2.diff

messages: + msg253493
2015-09-24 11:38:45hayposetmessages: + msg251510
2015-09-24 11:38:18hayposetmessages: + msg251509
2015-09-22 21:44:54hayposetnosy: + haypo
2014-02-05 20:08:35serhiy.storchakasetmessages: + msg210346
2014-02-05 16:57:23r.david.murraysetversions: + Python 3.5, - Python 2.7, Python 3.3, Python 3.4
type: resource usage -> enhancement

nosy: + ned.deily, serhiy.storchaka
title: ctypes.create_string_buffer creates reference cycles -> Replace uuid ctypes usage with an extension module.
messages: + msg210335
stage: patch review
2014-02-05 16:16:56gustavosetfiles: + uuid-no-ctypes.diff
keywords: + patch
messages: + msg210331
2014-02-05 15:44:17r.david.murraysettitle: uuid.uuid4().hex generates garbage when ctypes available -> ctypes.create_string_buffer creates reference cycles
nosy: + amaury.forgeotdarc, belopolsky, meador.inge

messages: + msg210328

components: + ctypes, - Library (Lib)
2014-02-05 14:58:29gustavosetmessages: + msg210322
2014-02-05 14:28:22gustavosetmessages: + msg210320
2014-02-05 14:10:44r.david.murraysetnosy: + r.david.murray
messages: + msg210316
2014-02-05 11:52:23gustavocreate