classification
Title: ctypes: unions as arguments
Type: crash Stage: patch review
Components: ctypes, Documentation Versions: Python 3.2, Python 3.3, Python 3.4, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: vinay.sajip Nosy List: Arfrever, alex, ammar2, arigo, docs@python, eli.bendersky, meador.inge, python-dev, theller, vinay.sajip, vstinner
Priority: normal Keywords: patch

Created on 2012-11-29 04:24 by arigo, last changed 2020-01-12 20:55 by vinay.sajip.

Files
File name Uploaded Description Edit
test184_lib.tgz arigo, 2012-11-29 04:24
Pull Requests
URL Status Linked Edit
PR 15839 vinay.sajip, 2019-09-10 11:38
PR 16430 closed vinay.sajip, 2019-09-26 16:48
PR 16799 merged vinay.sajip, 2019-10-15 08:36
PR 17016 merged vinay.sajip, 2019-10-31 13:01
PR 17017 merged vinay.sajip, 2019-10-31 13:09
PR 17064 merged ammar2, 2019-11-05 18:55
PR 17960 merged vinay.sajip, 2020-01-12 08:27
PR 17964 merged miss-islington, 2020-01-12 11:23
PR 17970 merged vinay.sajip, 2020-01-12 20:29
Messages (20)
msg176627 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2012-11-29 04:24
ctypes pretends to support passing arguments to C functions that are unions (not pointers to unions), but that's a lie.  In fact, the underlying libffi does not support it.  The attached example misbehaves on Linux x86-64.
msg183461 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-03-04 14:54
A minimal fix would be to update the documentation.

A more comprehensive fix would be to tweak ctypes to reject unions and bit-fields when running on non-x86 (does this work for ARM and other non-Intel archs?)

An even more comprehensive fix would be to make this work, but that would require forking libffi and should presumably be first implemented upstream.
msg183464 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2013-03-04 15:12
See also http://bugs.python.org/issue16576.
msg183817 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-03-09 13:56
New changeset 0acd9408b6f1 by Eli Bendersky in branch '3.2':
Add warning in ctypes documentation for #16575 and #16576
http://hg.python.org/cpython/rev/0acd9408b6f1

New changeset bfc159f8e4b4 by Eli Bendersky in branch '3.3':
Add warning in ctypes documentation for #16575 and #16576
http://hg.python.org/cpython/rev/bfc159f8e4b4

New changeset 502624235c7b by Eli Bendersky in branch 'default':
Add warning in ctypes documentation for #16575 and #16576
http://hg.python.org/cpython/rev/502624235c7b
msg183819 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-03-09 13:57
New changeset eece32440a52 by Eli Bendersky in branch '2.7':
Add warning in ctypes documentation for #16575 and #16576
http://hg.python.org/cpython/rev/eece32440a52
msg186689 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-04-13 03:12
I've opened a libffi issue in an attempt to get this fixed upstream:

  https://github.com/atgreen/libffi/issues/33
msg351454 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-09-09 13:11
Link to issue has changed to:

https://github.com/libffi/libffi/issues/33
msg355741 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-10-31 08:04
New changeset 79d4ed102a5069c6cebaed2627cb1645637f0429 by Vinay Sajip in branch 'master':
bpo-16575: Add checks for unions passed by value to functions. (GH-16799)
https://github.com/python/cpython/commit/79d4ed102a5069c6cebaed2627cb1645637f0429
msg355751 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-10-31 13:34
New changeset 952899704800a5aa438e84d50f0b6fc498c72e89 by Vinay Sajip in branch '3.8':
[3.8] bpo-16575: Add checks for unions passed by value to functions. (GH-16799) (GH-17016)
https://github.com/python/cpython/commit/952899704800a5aa438e84d50f0b6fc498c72e89
msg355754 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-10-31 14:15
New changeset 0118d109d54bf75c99a8b0fa9aeae1a478ac4b7e by Vinay Sajip in branch '3.7':
[3.7] bpo-16575: Add checks for unions passed by value to functions. (GH-16799) (GH-17017)
https://github.com/python/cpython/commit/0118d109d54bf75c99a8b0fa9aeae1a478ac4b7e
msg355993 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-05 01:29
This change introduced a reference leak on Windows. Example on 3.7:

https://buildbot.python.org/all/#/builders/132/builds/645

test_ctypes leaked [174, 174, 174] references, sum=522
test_ctypes leaked [76, 77, 77] memory blocks, sum=230
msg355994 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-05 01:31
Same issue on x86 Gentoo Refleaks 3.7:

test_ctypes leaked [174, 174, 174] references, sum=522
test_ctypes leaked [76, 78, 76] memory blocks, sum=230

https://buildbot.python.org/all/#/builders/114/builds/631
msg356053 - (view) Author: Ammar Askar (ammar2) * (Python triager) Date: 2019-11-05 18:58
Opened https://github.com/python/cpython/pull/17064 to fix this. Essentially it's a tiny little oversight in the back-porting. In the 3.7 branch, we perform an attribute lookup for `from_param` before the union checking code, so we must remember to DECREF it. This is unlike master where the attribute lookup happens after the union checking code.



λ win32\python_d.exe -m test -R 3:3 test_ctypes
0:00:00 Run tests sequentially
0:00:00 [1/1] test_ctypes
beginning 6 repetitions
123456
......

== Tests result: SUCCESS ==

1 test OK.

Total duration: 14.8 sec
Tests result: SUCCESS
msg356141 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-11-06 15:40
New changeset 484edbf9bf1a9e6bae0fcb10a0c165b89ea79295 by Vinay Sajip (Ammar Askar) in branch '3.7':
bpo-16575: Fix refleak on passing unions in ctypes (GH-17064)
https://github.com/python/cpython/commit/484edbf9bf1a9e6bae0fcb10a0c165b89ea79295
msg356147 - (view) Author: Ammar Askar (ammar2) * (Python triager) Date: 2019-11-06 17:35
Will close after https://buildbot.python.org/all/#/builders?tags=%2Brefleak&tags=%2B3.7 go back to green.
msg356230 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-08 08:14
3.7 refleaks buildbots pass again (ignoring a few warnings).
msg359834 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2020-01-12 07:44
It seems that notwithstanding the issues identified in the original bug report, calls by value of structs containing unions are being used out there in the wild. Examples:

The comtypes library (Windows):
https://github.com/python/cpython/pull/16799#issuecomment-573318591

The ctypesgen library, which is used by other projects across platforms:
https://github.com/davidjamesca/ctypesgen/issues/77

Linux examples:
https://github.com/waveform80/picamera/issues/604
https://github.com/hcpl/xkbgroup/issues/15
https://github.com/enkore/i3pystatus/issues/771

So, it would seem that the simplest course of action is to disable the checks for now. The checks will be commented out and this issue and the related issue bpo-16576 will be reopened.
msg359836 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2020-01-12 08:54
New changeset c12440c371025bea9c3bfb94945f006c486c2c01 by Vinay Sajip in branch 'master':
bpo-16575: Disabled checks for union types being passed by value. (GH-17960)
https://github.com/python/cpython/commit/c12440c371025bea9c3bfb94945f006c486c2c01
msg359844 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2020-01-12 11:41
New changeset eb9ba2f66df2195a9c6295e73ab3d545a1445f05 by Vinay Sajip (Miss Islington (bot)) in branch '3.8':
bpo-16575: Disabled checks for union types being passed by value. (GH-17960) (GH-17964)
https://github.com/python/cpython/commit/eb9ba2f66df2195a9c6295e73ab3d545a1445f05
msg359876 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2020-01-12 20:55
New changeset 9dbf5d3bc2033940cdca35440cf08814544f81e4 by Vinay Sajip in branch '3.7':
[3.7] bpo-16575: Disabled checks for union types being passed by value. (GH-17960) (GH-17970)
https://github.com/python/cpython/commit/9dbf5d3bc2033940cdca35440cf08814544f81e4
History
Date User Action Args
2020-01-12 20:55:57vinay.sajipsetmessages: + msg359876
2020-01-12 20:29:13vinay.sajipsetpull_requests: + pull_request17378
2020-01-12 11:41:11vinay.sajipsetmessages: + msg359844
2020-01-12 11:23:28miss-islingtonsetpull_requests: + pull_request17373
2020-01-12 08:54:05vinay.sajipsetmessages: + msg359836
2020-01-12 08:30:24vinay.sajiplinkissue26628 dependencies
2020-01-12 08:27:48vinay.sajipsetstage: patch review
pull_requests: + pull_request17368
2020-01-12 07:44:02vinay.sajipsetstatus: closed -> open
resolution: fixed ->
messages: + msg359834

stage: resolved -> (no value)
2019-11-08 08:14:56vstinnersetresolution: fixed
messages: + msg356230
2019-11-07 02:39:27ammar2setstatus: open -> closed
stage: patch review -> resolved
2019-11-06 17:35:07ammar2setmessages: + msg356147
2019-11-06 15:40:14vinay.sajipsetmessages: + msg356141
2019-11-05 18:58:13ammar2setnosy: + ammar2
messages: + msg356053
2019-11-05 18:55:28ammar2setstage: resolved -> patch review
pull_requests: + pull_request16572
2019-11-05 01:31:11vstinnersetmessages: + msg355994
2019-11-05 01:29:42vstinnersetstatus: closed -> open

nosy: + vstinner
messages: + msg355993

resolution: fixed -> (no value)
2019-10-31 14:17:04vinay.sajipsetstatus: open -> closed
assignee: docs@python -> vinay.sajip
resolution: fixed
stage: patch review -> resolved
2019-10-31 14:15:18vinay.sajipsetmessages: + msg355754
2019-10-31 13:34:11vinay.sajipsetmessages: + msg355751
2019-10-31 13:09:54vinay.sajipsetpull_requests: + pull_request16536
2019-10-31 13:01:34vinay.sajipsetpull_requests: + pull_request16535
2019-10-31 08:04:02vinay.sajipsetmessages: + msg355741
2019-10-15 08:36:48vinay.sajipsetpull_requests: + pull_request16355
2019-09-26 16:48:13vinay.sajipsetpull_requests: + pull_request16009
2019-09-10 11:38:40vinay.sajipsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request15485
2019-09-09 13:11:43vinay.sajipsetmessages: + msg351454
2017-02-23 08:18:33vinay.sajipsetnosy: + vinay.sajip
2016-06-23 09:30:48berker.peksaglinkissue14354 superseder
2013-04-13 03:12:03eli.benderskysetmessages: + msg186689
2013-03-18 12:26:28eli.benderskysetnosy: + theller
2013-03-09 13:57:30python-devsetmessages: + msg183819
2013-03-09 13:56:20python-devsetnosy: + python-dev
messages: + msg183817
2013-03-04 15:12:57arigosetmessages: + msg183464
2013-03-04 14:54:53eli.benderskysetassignee: docs@python
components: + Documentation
versions: + Python 3.3, Python 3.4
nosy: + eli.bendersky, docs@python

messages: + msg183461
stage: needs patch
2012-12-14 08:30:56Arfreversetnosy: + Arfrever
2012-12-11 06:20:05meador.ingesetnosy: + meador.inge
2012-11-29 04:49:13arigosettype: crash
2012-11-29 04:33:04alexsetnosy: + alex
2012-11-29 04:24:04arigocreate