Issue42880
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2021-01-10 07:41 by lazymio, last changed 2022-04-11 14:59 by admin.
Messages (13) | |||
---|---|---|---|
msg384756 - (view) | Author: Ziqiao Kong (lazymio) | Date: 2021-01-10 07:41 | |
Hello! Thanks for your contribution on porting python to native Apple Silicon. I have noticed that there is an issue https://bugs.python.org/issue41100 and corresponding Github PR https://github.com/python/cpython/pull/22855 stating that variadic functions ABI has been corrected. However, during our test, it still doesn't work on Apple Silicon with brew-installed python 3.9.1 and python 3..10.0a4 built from sources. The details are as follows: A x86_64 Mojave Macmini with python 3.8: ``` /tmp $ python3 --version Python 3.8.6 /tmp $ uname -a Darwin *** 18.7.0 Darwin Kernel Version 18.7.0: Mon Apr 27 20:09:39 PDT 2020; root:xnu-4903.278.35~1/RELEASE_X86_64 x86_64 /tmp $ cat main.c #include <stdarg.h> #include <stdio.h> #include <stdint.h> void test(uint64_t end, ...){ int a; va_list valist; va_start(valist, end); a = va_arg(valist, int); va_end(valist); printf("%d\n", a); return; } /tmp $ cc -shared -g main.c -o ./main.dylib /tmp $ cat test_main.py from ctypes import * d = CDLL("./main.dylib") d.test.restype = None d.test(c_uint64(12), c_int(34)) /tmp $ python3 test_main.py 34 /tmp $ ``` An M1 Macbook Pro with brew-installed python 3.9.1 ``` kabeor@kamino /tmp % python3 --version Python 3.9.1 kabeor@kamino /tmp % cat main.c #include <stdarg.h> #include <stdio.h> #include <stdint.h> void test(uint64_t end, ...){ int a; va_list valist; va_start(valist, end); a = va_arg(valist, int); va_end(valist); printf("%d\n", a); return; } kabeor@kamino /tmp % cc -shared -g main.c -o ./main.dylib kabeor@kamino /tmp % cat test_main.py from ctypes import * d = CDLL("./main.dylib") d.test.restype = None d.test(c_uint64(12), c_int(34)) kabeor@kamino /tmp % python3 test_main.py 48144104 kabeor@kamino /tmp % ``` An M1 Macbook Pro with python 3.10.0a4 built from Github release tarball ``` kabeor@kamino cpython-3.10.0a4 % ./python.exe --version Python 3.10.0a4 kabeor@kamino cpython-3.10.0a4 % cp /tmp/main.c ./ kabeor@kamino cpython-3.10.0a4 % cp /tmp/test_main.py ./ kabeor@kamino cpython-3.10.0a4 % ./python.exe --version Python 3.10.0a4 kabeor@kamino cpython-3.10.0a4 % cat ./main.c #include <stdarg.h> #include <stdio.h> #include <stdint.h> void test(uint64_t end, ...){ int a; va_list valist; va_start(valist, end); a = va_arg(valist, int); va_end(valist); printf("%d\n", a); return; } kabeor@kamino cpython-3.10.0a4 % cc -shared -g main.c -o ./main.dylib kabeor@kamino cpython-3.10.0a4 % cat test_main.py from ctypes import * d = CDLL("./main.dylib") d.test.restype = None d.test(c_uint64(12), c_int(34)) kabeor@kamino cpython-3.10.0a4 % ./python.exe test_main.py 3 kabeor@kamino cpython-3.10.0a4 % ``` Thanks in advance! |
|||
msg384771 - (view) | Author: Ned Deily (ned.deily) * ![]() |
Date: 2021-01-10 18:01 | |
I tried your test programs on a M1 Mac running macOS 11.1 with the latest macOS universal2 Pythons available from python.org (https://www.python.org/downloads/), 3.9.1 and 3.10.0a4, and both produced the same (correct) result as in your 3.8 example. Are you sure the brew Pythons are using the macOS 11 system libffi and not their own or older version? |
|||
msg384772 - (view) | Author: Ned Deily (ned.deily) * ![]() |
Date: 2021-01-10 18:17 | |
(I also get *34* when building v3.10.0a4 from source on the M1 with 11.1 and Xcode 12.3.) |
|||
msg384773 - (view) | Author: Ziqiao Kong (lazymio) | Date: 2021-01-10 18:20 | |
Thanks for your reply. What shall I do to confirm the version of libffi? By otool? ________________________________ From: report=bugs.python.org@roundup.psfhosted.org <report=bugs.python.org@roundup.psfhosted.org> on behalf of Ned Deily <report@bugs.python.org> Sent: Monday, January 11, 2021 2:17:53 AM To: ziqiaokong@gmail.com <ziqiaokong@gmail.com> Subject: [issue42880] ctypes: variadic function call still doesn't work on Apple Silicon Ned Deily <nad@python.org> added the comment: (I also get *34* when building v3.10.0a4 from source on the M1 with 11.1 and Xcode 12.3.) ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue42880> _______________________________________ |
|||
msg384774 - (view) | Author: Ned Deily (ned.deily) * ![]() |
Date: 2021-01-10 18:24 | |
Yes, otool should work. Something like: $ otool -L $(python3.9 -c 'import _ctypes;print(_ctypes.__file__)') /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/_ctypes.cpython-39-darwin.so: /usr/lib/libffi.dylib (compatibility version 1.0.0, current version 27.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.0.0) /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1770.106.0) |
|||
msg384795 - (view) | Author: Ziqiao Kong (lazymio) | Date: 2021-01-11 06:19 | |
Seems that we are getting the same libffi.dylib. kabeor@kamino cpython-3.10.0a4 % otool -L $(./python.exe -c 'import _ctypes;print(_ctypes.__file__)') /Users/kabeor/cpython-3.10.0a4/build/lib.macosx-11.0-arm64-3.10/_ctypes.cpython-310-darwin.so: /usr/lib/libffi.dylib (compatibility version 1.0.0, current version 27.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1) |
|||
msg384796 - (view) | Author: Ziqiao Kong (lazymio) | Date: 2021-01-11 06:24 | |
BTW, uname for the M1 MBP. ``` kabeor@kamino cpython-3.10.0a4 % uname -a Darwin kamino.lan 20.1.0 Darwin Kernel Version 20.1.0: Sat Oct 31 00:07:10 PDT 2020; root:xnu-7195.50.7~2/RELEASE_ARM64_T8101 arm6464 ``` |
|||
msg384844 - (view) | Author: Ned Deily (ned.deily) * ![]() |
Date: 2021-01-11 16:10 | |
> Seems that we are getting the same libffi.dylib. I suppose it is still possible that another libffi is being found first. To be absolutely sure, you could ask dyld to print the loaded libraries: $ DYLD_PRINT_LIBRARIES=1 python3.9 test_main.py [...] dyld: loaded: <003A027D-9CE3-3794-A319-88495844662D> /usr/lib/system/libxpc.dylib dyld: loaded: <7FBC5290-B2B3-3312-B69A-77378C539520> /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/_ctypes.cpython-39-darwin.so dyld: loaded: <943473A5-A82B-3E21-8EB3-0BBDF5E605AA> /usr/lib/libffi.dylib dyld: loaded: <58910956-4F2F-363A-80E5-D5E1C71DD83E> /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/_struct.cpython-39-darwin.so dyld: loaded: <3444F392-DB43-3AEF-8596-2BFFBBD8EC0C> ./main.dylib 34 But, more importantly, the uname -a output for my M1 is: Darwin ankl.local 20.2.0 Darwin Kernel Version 20.2.0: Wed Dec 2 20:40:21 PST 2020; root:xnu-7195.60.75~1/RELEASE_ARM64_T8101 x86_64 Perhaps you have not updated to macOS 11.1 yet? By the way, in your original message's last example, is the output really "3"? |
|||
msg384880 - (view) | Author: Ziqiao Kong (lazymio) | Date: 2021-01-12 03:37 | |
Sorry that our test machine failed to boot due to some firmware problem when upgrading to 11.1 yesterday. I will re-run my tests after the upgrade gets done. Also, I'm sure the last output is "3" in my first message. |
|||
msg384900 - (view) | Author: Ziqiao Kong (lazymio) | Date: 2021-01-12 08:10 | |
Hello, we do a clean reinstallation of Big Sur 11.1. The problem still exists: with brew-installed python3.9 ``` qiling@kamino /tmp % DYLD_PRINT_LIBRARIES=1 python3.9 test_main.py [...] dyld: loaded: <52918C9B-7E0B-3852-AB19-F74846BCAAF8> /Users/qiling/brew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/_heapq.cpython-39-darwin.so dyld: loaded: <A53A85D2-2C36-32E5-A44A-3B7C061CE7BE> /Users/qiling/brew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/_ctypes.cpython-39-darwin.so dyld: loaded: <EFF19129-E929-3E58-A6C4-16FB9F26120C> /usr/lib/libffi.dylib dyld: loaded: <656F7BE5-2BA6-3FE9-BD31-77ED659B6194> /Users/qiling/brew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/_struct.cpython-39-darwin.so dyld: loaded: <C23349E3-BDF1-3964-A151-947F47728CC0> ./main.dylib 50900232 qiling@kamino /tmp % DYLD_PRINT_LIBRARIES=1 python3.9 test_main.py [...] dyld: loaded: <52918C9B-7E0B-3852-AB19-F74846BCAAF8> /Users/qiling/brew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/_heapq.cpython-39-darwin.so dyld: loaded: <A53A85D2-2C36-32E5-A44A-3B7C061CE7BE> /Users/qiling/brew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/_ctypes.cpython-39-darwin.so dyld: loaded: <EFF19129-E929-3E58-A6C4-16FB9F26120C> /usr/lib/libffi.dylib dyld: loaded: <656F7BE5-2BA6-3FE9-BD31-77ED659B6194> /Users/qiling/brew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/_struct.cpython-39-darwin.so dyld: loaded: <C23349E3-BDF1-3964-A151-947F47728CC0> ./main.dylib 86961416 ``` even the output differs between different runs... uname -a ``` Darwin kamino.lan 20.2.0 Darwin Kernel Version 20.2.0: Wed Dec 2 20:40:21 PST 2020; root:xnu-7195.60.75~1/RELEASE_ARM64_T8101 arm64 ``` btw, the python 3.10.0a4 built from sources: ``` qiling@kamino cpython-3.10.0a4 % DYLD_PRINT_LIBRARIES=1 ./python.exe test_main.py dyld: loaded: <483C8366-9BE1-3A03-B41A-4A526F6A6F91> /Users/qiling/cpython-3.10.0a4/./python.exe [...] dyld: loaded: <97E08C95-0A13-3030-AC8D-3330CB10AD0B> /Users/qiling/cpython-3.10.0a4/build/lib.macosx-11.1-arm64-3.10/_ctypes.cpython-310-darwin.so dyld: loaded: <EFF19129-E929-3E58-A6C4-16FB9F26120C> /usr/lib/libffi.dylib dyld: loaded: <9CA1E334-8C96-362A-B21F-9A2C43ACF58D> /Users/qiling/cpython-3.10.0a4/build/lib.macosx-11.1-arm64-3.10/_struct.cpython-310-darwin.so dyld: loaded: <AF35C7F2-8D8D-3D1B-A52D-9A37263161BA> ./main.dylib 3 qiling@kamino cpython-3.10.0a4 % DYLD_PRINT_LIBRARIES=1 ./python.exe test_main.py dyld: loaded: <483C8366-9BE1-3A03-B41A-4A526F6A6F91> /Users/qiling/cpython-3.10.0a4/./python.exe [...] dyld: loaded: <97E08C95-0A13-3030-AC8D-3330CB10AD0B> /Users/qiling/cpython-3.10.0a4/build/lib.macosx-11.1-arm64-3.10/_ctypes.cpython-310-darwin.so dyld: loaded: <EFF19129-E929-3E58-A6C4-16FB9F26120C> /usr/lib/libffi.dylib dyld: loaded: <9CA1E334-8C96-362A-B21F-9A2C43ACF58D> /Users/qiling/cpython-3.10.0a4/build/lib.macosx-11.1-arm64-3.10/_struct.cpython-310-darwin.so dyld: loaded: <AF35C7F2-8D8D-3D1B-A52D-9A37263161BA> ./main.dylib 3 qiling@kamino cpython-3.10.0a4 % ``` |
|||
msg384901 - (view) | Author: Ronald Oussoren (ronaldoussoren) * ![]() |
Date: 2021-01-12 09:00 | |
AFAIK you should specify the types of the fixed arguments for variadic functions. The ABI for the M1 has a different calling convention of the variadic arguments, the ctypes implementation counts how many variadic arguments there are based on the specified number of fixed arguments. |
|||
msg384902 - (view) | Author: Ziqiao Kong (lazymio) | Date: 2021-01-12 09:09 | |
I retry my cases with types specified and it works indeed. Maybe the documents should be updated for this case. Thanks a lot! |
|||
msg384934 - (view) | Author: Ronald Oussoren (ronaldoussoren) * ![]() |
Date: 2021-01-12 13:27 | |
I agree that this should be mentioned in the documentation for ctypes. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:59:40 | admin | set | github: 87046 |
2021-01-12 13:27:45 | ronaldoussoren | set | assignee: docs@python components: + Documentation nosy: + docs@python |
2021-01-12 13:27:23 | ronaldoussoren | set | messages: + msg384934 |
2021-01-12 09:09:39 | lazymio | set | messages: + msg384902 |
2021-01-12 09:00:10 | ronaldoussoren | set | messages: + msg384901 |
2021-01-12 08:10:58 | lazymio | set | messages: + msg384900 |
2021-01-12 03:37:38 | lazymio | set | messages: + msg384880 |
2021-01-11 16:10:47 | ned.deily | set | messages: + msg384844 |
2021-01-11 06:24:26 | lazymio | set | messages: + msg384796 |
2021-01-11 06:19:12 | lazymio | set | messages: + msg384795 |
2021-01-10 18:24:32 | ned.deily | set | messages: + msg384774 |
2021-01-10 18:20:25 | lazymio | set | messages: + msg384773 |
2021-01-10 18:17:53 | ned.deily | set | messages: + msg384772 |
2021-01-10 18:01:31 | ned.deily | set | messages: + msg384771 |
2021-01-10 17:49:48 | ned.deily | set | nosy:
+ ronaldoussoren, ned.deily components: + macOS |
2021-01-10 07:41:58 | lazymio | create |