classification
Title: Consider supporting emscripten/webassembly as a build target
Type: enhancement Stage: patch review
Components: Build Versions: Python 3.11
process
Status: open Resolution:
Dependencies: 33393 41498 45881 45886 45898 Superseder:
Assigned To: Nosy List: Roman Yurchak, Simon Biggs, barry, brett.cannon, christian.heimes, corona10, erlendaasland, ethan smith, mcepl, miss-islington, pmpp, serhiy.storchaka, xtreak
Priority: normal Keywords: patch

Created on 2020-04-14 07:11 by Simon Biggs, last changed 2021-12-02 17:43 by christian.heimes.

Pull Requests
URL Status Linked Edit
PR 29771 merged christian.heimes, 2021-11-25 10:07
PR 29784 merged christian.heimes, 2021-11-25 20:55
PR 29789 merged christian.heimes, 2021-11-26 08:49
PR 29794 merged christian.heimes, 2021-11-26 12:52
PR 29795 merged christian.heimes, 2021-11-26 13:10
PR 29834 merged erlendaasland, 2021-11-29 10:23
PR 29842 merged christian.heimes, 2021-11-29 13:37
PR 29872 merged christian.heimes, 2021-12-01 08:05
PR 29873 merged christian.heimes, 2021-12-01 08:10
PR 29884 merged christian.heimes, 2021-12-01 21:02
PR 29887 merged christian.heimes, 2021-12-02 08:19
PR 29892 open ethan smith, 2021-12-02 13:16
PR 29893 merged christian.heimes, 2021-12-02 14:28
Messages (29)
msg366369 - (view) Author: Simon Biggs (Simon Biggs) Date: 2020-04-14 07:11
Since asm.js came on the scene, and now Web Assembly people have created CPython patches to support building CPython with emscripten. See:

* https://github.com/PeachPy/EmCPython -- Python 2.7
* https://github.com/dgym/cpython-emscripten/tree/master/3.5.2/patches -- Python 3.5.2
* https://github.com/iodide-project/pyodide/tree/master/cpython/patches -- Python 3.7.4

To ease the compiling of CPython with emscripten it would be helpful if patches that achieved these ends for the compiling to Web Assembly with emscripten were built into the upstream source repository itself.

If web assembly were to became a supported compilation target of the upstream CPython repository this would significantly reduce the friction of allowing CPython, and the latest CPython, to become a language readily usable within the browser.

Cheers,
Simon
msg366374 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-04-14 08:14
Do you want to provide a pull request?
msg366375 - (view) Author: pmp-p (pmpp) * Date: 2020-04-14 08:58
you can add 
* https://github.com/pmp-p/pydk/tree/master/sources.em/Python-3.8.0b4.patchset -- Python 3.8.x 

(wasm not asm.js, clang-10+ required)

demo https://pmp-p.github.io/python-next/test.html

CPython can already run in the browser with very little patching, but major issues are :
 
 - asyncify'ing the whole wasm VM to have pre-emption over cPython's one to prevent blocking I/O slows down things *a lot* (10x)
=> (very?) bad user experience.

 - the size of vm + stdlib  ~ 30 MiB and wasm compilation time.
=> bad user experience on first load or slow connexion.

 - the lack of threading in wasm MinimumViableProduct specification (but this is the browser standard for now), that leads to rewrite bits of stdlib ( like eg asyncio module )
=> adding more maintenance burden on stdlib (!)


i tested them all and my personnal opinion is : I can see no use case that would favour "stock" cPython wasm versus a blazing fast MicroPytho (or pycopy) wasm flavour or supercharged full stack pyodide.
msg366379 - (view) Author: Simon Biggs (Simon Biggs) Date: 2020-04-14 12:39
Hi pmp-p and Serhiy,

I'd be more than happy to attempt a pull request, but I imagine a change
such as this needs to be discussed first, trying not to "rush to make a
patch" (https://www.youtube.com/watch?v=voXVTjwnn-U&feature=youtu.be&t=2546).
Also, I doubt I will do a good job of it... but I am more than happy to try.

A note regarding "supercharged full stack pyodide", potentially without
efforts such as upstreaming into CPython and emscripten the relevant
patches, that supercharged full stack may just unfortunately stagnate. See
https://github.com/iodide-project/pyodide/issues/635#issuecomment-613408912

With respect to blocking when running Python as WASM, I have found running
the WebAssembly CPython within a webworker and signalling data back and
forth causes there to be no UI issues. It ends up being quite a neat set
up. Main down side right now however is the set up is currently going
stale, hence me believing reaching out like this is in the best interests
of Python going forward.

Cheers,
Simon

On Tue, 14 Apr 2020 at 18:58, pmp-p <report@bugs.python.org> wrote:

>
> pmp-p <pmpp.pub@gmail.com> added the comment:
>
> you can add
> *
> https://github.com/pmp-p/pydk/tree/master/sources.em/Python-3.8.0b4.patchset
> -- Python 3.8.x
>
> (wasm not asm.js, clang-10+ required)
>
> demo https://pmp-p.github.io/python-next/test.html
>
> CPython can already run in the browser with very little patching, but
> major issues are :
>
>  - asyncify'ing the whole wasm VM to have pre-emption over cPython's one
> to prevent blocking I/O slows down things *a lot* (10x)
> => (very?) bad user experience.
>
>  - the size of vm + stdlib  ~ 30 MiB and wasm compilation time.
> => bad user experience on first load or slow connexion.
>
>  - the lack of threading in wasm MinimumViableProduct specification (but
> this is the browser standard for now), that leads to rewrite bits of stdlib
> ( like eg asyncio module )
> => adding more maintenance burden on stdlib (!)
>
>
> i tested them all and my personnal opinion is : I can see no use case that
> would favour "stock" cPython wasm versus a blazing fast MicroPytho (or
> pycopy) wasm flavour or supercharged full stack pyodide.
>
> ----------
> nosy: +pmpp
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue40280>
> _______________________________________
>
msg369598 - (view) Author: (Beuc) * Date: 2020-05-22 11:21
I've been maintaining a Python Emscripten build for the Ren'Py (game engine) web port:
https://github.com/python-emscripten/python
https://renpy.beuc.net/

I recently tackled Python3 with a minimal/embeddable approach and checking the other ports already pointed in the discussion:
https://github.com/python-emscripten/python/tree/trunk/3.8/
(2 patches, and a short pyconfig.h fix-up)

There is also a Cython module to use the Emscripten C API.

Here's a demo at:
https://www.beuc.net/python-emscripten/demo/3/
(hello-world size: 3MB, with a few common modules: 4MB)

I can provide a pull request with a first few core changes.
Should this be done on github?

-

cross-compilation handling appears to follow an incorrect logic, in particular by querying 'dpkg' or parsing compiler output to detect include paths -- it is the (cross-)compiler's responsibility to provide the system paths, and detecting them manually causes conflicts. I had to patch setup.py. Let me know if I missed something.

Usually cross-compiling is triggered by non-matching build-type/host-type. Here cross-compilation logic is apparently triggered when exporting _PYTHON_HOST_PLATFORM=xxx manually (it's in the Makefile but not exported, and breaks normal build when exported). Is this the way it's meant to be used?
msg394081 - (view) Author: Ethan Smith (ethan smith) * Date: 2021-05-20 23:49
I think the first thing we should do is figure out whether we want to support Emscripten or WASI (or both).

Emscripten uses Javascript polyfills for some syscalls, while WASI makes direct calls the VM it is running in. They both can use WebAssembly for executing the code. This means Emscripten has wider API support, but WASI is lighter weight in many ways.

I think starting with patches to support Emscripten would be best, as it is easier to target, then add support for WASI later.

I think supporting WASI has a lot of value, because it can be run deterministically, which would be great for data science (Imagine a jupyter notebook that runs the same everywhere!)

One issue with WASI, and may be an issue with Emscripten, is threads. In 3.8 (or 3.9?) threadless builds were removed. However, WebAssembly's threading API is not really meant to emulate pthread, and SharedArrayBuffer, the primitive it is built on, is disabled in several browsers due to Spectre concerns.

Would patches to re-add a threadless build mode be accepted?
msg406985 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-25 10:23
I have added wasm32/wasm64 architectures with emscripten/wasi operating system as cross-build targets. The values are based on Rust targets: 

$ rustc --print target-list | grep wasm
wasm32-unknown-emscripten
wasm32-unknown-unknown
wasm32-wasi
wasm64-unknown-unknown

wasm (WebAssembly) is "native instruction set" for the JavaScript VM while wasi or emscripten provide operating system facilities like memory management and I/O.
msg406986 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-25 10:50
Our config.sub is recent enough and has support for wasm32, wasm64, wasi, and emscripten:

$ grep was[mi] config.sub 
                        | wasm32 | wasm64 \
             | midnightbsd* | amdhsa* | unleashed* | emscripten* | wasi* \
msg407011 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2021-11-25 18:39
LLVM considers `was32-wasi` an alias for `wasm32-unknown-wasi`. Verified on the WebAssembly Discover server at https://discord.com/channels/453584038356058112/596492540388179976/898618010221310062.
msg407012 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2021-11-25 18:42
My last message had a couple of typos; should have been `wasm32-wasi` and "Discord", not "Discovery".
msg407013 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2021-11-25 18:48
Do we need to care about our `config.guess` being updated as well? This is a totally ignorant question based on https://github.com/WebAssembly/wasi-sdk#notes-for-autoconf mentioning `config.guess`.
msg407014 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2021-11-25 18:49
To help keep links up-to-date, Pyodide now lives at:

https://github.com/pyodide/pyodide/tree/main/cpython
msg407015 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-25 19:29
Our config.sub agrees with LLVM:

$ ./config.sub wasm32-wasi
wasm32-unknown-wasi

The config.sub and config.guess scripts in main are recent enough for wasm. Just to be sure I created https://github.com/python/cpython/pull/29781 and plan to backport the changeset to 3.10 and 3.9. It's generally safe to update the files to latest version.
msg407018 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-25 20:24
New changeset 1052a39b7603e4d8401a5987af0c36f4a1d0b1e4 by Christian Heimes in branch 'main':
bpo-40280: Add wasm cross build targets (GH-29771)
https://github.com/python/cpython/commit/1052a39b7603e4d8401a5987af0c36f4a1d0b1e4
msg407020 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-25 21:48
I have uploaded my config.site override to https://gist.github.com/tiran/5ccffa28723d3e4739db848451bd9efa . It contains overrides based on pyodide and overrides for new features.

I'm also getting this error with emscripten 2.0.13. _sys_shutdown is the syscall for shutdown(2) used by the socket module.

error: undefined symbol: __sys_shutdown (referenced by top-level compiled C/C++ code)
warning: Link with `-s LLD_REPORT_UNDEFINED` to get more information on undefined symbols
warning: To disable errors for undefined symbols use `-s ERROR_ON_UNDEFINED_SYMBOLS=0`
warning: ___sys_shutdown may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
Error: Aborting compilation due to previous errors
msg407021 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-25 21:51
https://github.com/emscripten-core/emscripten/issues/13393
msg407030 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-26 07:26
New changeset ee1e2c604c8a66a407116d9c3e589ab0b9580c54 by Christian Heimes in branch 'main':
bpo-40280: Use Setup.stdlib static for wasm builds (GH-29784)
https://github.com/python/cpython/commit/ee1e2c604c8a66a407116d9c3e589ab0b9580c54
msg407046 - (view) Author: Roman Yurchak (Roman Yurchak) Date: 2021-11-26 12:46
Thanks a lot for working on this!

> _sys_shutdown is the syscall for shutdown(2) used by the socket module.

Yes, the issue with Emscripten is that a number of system calls are either not implemented or implemented but not tested. See a list we are using in https://github.com/pyodide/pyodide/blob/main/cpython/pyconfig.undefs.h (though things might have improved since it was created).

FYI, with Emscripten, the list of CPython unit tests that are currently skipped (as of Python 3.9.5) is in https://github.com/pyodide/pyodide/blob/main/src/tests/python_tests.txt some of those are due to browser VM limitations (e.g. virtual filestem by Emscripten that's not fully POSIX compliant, no processes, no sockets, async only via the browser event loop etc), others because we are not yet using threading since not all browsers support it, and some failures probably need more investigation. 

Also opened https://github.com/pyodide/pyodide/issues/2000 . Let us know if there is anything we can do help with this effort.
msg407048 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-26 13:30
New changeset d224e769b83a6a027bec8e21ee62db2b96d5af8e by Christian Heimes in branch 'main':
bpo-40280: clean and ignore .wasm files (GH-29794)
https://github.com/python/cpython/commit/d224e769b83a6a027bec8e21ee62db2b96d5af8e
msg407049 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-26 13:53
Thanks Roman, I replied on the pyodide issue tracker.
msg407052 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-26 14:17
New changeset 8caceb7a474bf32cddfd25fba25b531ff65f4365 by Christian Heimes in branch 'main':
bpo-40280: Add configure check for socket shutdown (GH-29795)
https://github.com/python/cpython/commit/8caceb7a474bf32cddfd25fba25b531ff65f4365
msg407080 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-26 18:36
New changeset 4ebde73b8e416eeb1fd5d2ca3283f7ddb534c5b1 by Christian Heimes in branch 'main':
bpo-40280: Move hard-coded feature checks to configure (GH-29789)
https://github.com/python/cpython/commit/4ebde73b8e416eeb1fd5d2ca3283f7ddb534c5b1
msg407264 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-29 11:39
New changeset b3f443a35e08a473d52868621e9b268259139cb2 by Erlend Egeberg Aasland in branch 'main':
bpo-40280: Disable unusable core extension modules on emscripten (GH-29834)
https://github.com/python/cpython/commit/b3f443a35e08a473d52868621e9b268259139cb2
msg407284 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-29 15:02
New changeset 6ac3c8a3140c17bd71ba98dfc5250c371101e77c by Christian Heimes in branch 'main':
bpo-40280: Emscripten systems use .wasm suffix by default (GH-29842)
https://github.com/python/cpython/commit/6ac3c8a3140c17bd71ba98dfc5250c371101e77c
msg407484 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-12-01 19:43
New changeset 9deb83468c56c7484645e6e3a6d0183cd6a0afd7 by Christian Heimes in branch 'main':
bpo-40280: Emscripten defaults to --with-ensurepip=no (GH-29873)
https://github.com/python/cpython/commit/9deb83468c56c7484645e6e3a6d0183cd6a0afd7
msg407498 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-12-01 22:16
New changeset 309110f37cdfc78d160ed08ae8faa6f6160ba87e by Christian Heimes in branch 'main':
bpo-40280: Emscripten with_ensurepip=no, second attempt (GH-29884)
https://github.com/python/cpython/commit/309110f37cdfc78d160ed08ae8faa6f6160ba87e
msg407519 - (view) Author: miss-islington (miss-islington) Date: 2021-12-02 09:17
New changeset cb2b3c8d3566ae46b3b8d0718019e1c98484589e by Christian Heimes in branch 'main':
bpo-40280: Emscripten has no support for subprocesses (GH-29872)
https://github.com/python/cpython/commit/cb2b3c8d3566ae46b3b8d0718019e1c98484589e
msg407524 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-12-02 11:19
New changeset cb8f491f46e262549f6c447b31625cab7c20a60a by Christian Heimes in branch 'main':
bpo-40280: Optimize ints and and startup on wasm (GH-29887)
https://github.com/python/cpython/commit/cb8f491f46e262549f6c447b31625cab7c20a60a
msg407541 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-12-02 17:43
New changeset a31173c5ceb1708df687f942d714bdecae7cb759 by Christian Heimes in branch 'main':
bpo-40280: Update what's new (GH-29893)
https://github.com/python/cpython/commit/a31173c5ceb1708df687f942d714bdecae7cb759
History
Date User Action Args
2021-12-02 17:43:05christian.heimessetmessages: + msg407541
2021-12-02 14:28:26christian.heimessetpull_requests: + pull_request28117
2021-12-02 13:16:06ethan smithsetpull_requests: + pull_request28116
2021-12-02 11:19:35christian.heimessetmessages: + msg407524
2021-12-02 09:17:46miss-islingtonsetnosy: + miss-islington
messages: + msg407519
2021-12-02 08:19:20christian.heimessetpull_requests: + pull_request28113
2021-12-01 22:16:35christian.heimessetmessages: + msg407498
2021-12-01 21:02:12christian.heimessetpull_requests: + pull_request28110
2021-12-01 19:43:34christian.heimessetmessages: + msg407484
2021-12-01 08:10:59christian.heimessetpull_requests: + pull_request28099
2021-12-01 08:05:36christian.heimessetpull_requests: + pull_request28098
2021-11-29 15:02:03christian.heimessetmessages: + msg407284
2021-11-29 13:37:54christian.heimessetpull_requests: + pull_request28071
2021-11-29 11:39:39christian.heimessetmessages: + msg407264
2021-11-29 10:23:23erlendaaslandsetnosy: + erlendaasland
pull_requests: + pull_request28065
2021-11-26 18:36:58christian.heimessetmessages: + msg407080
2021-11-26 14:17:03christian.heimessetmessages: + msg407052
2021-11-26 14:16:09corona10setnosy: + corona10
2021-11-26 13:53:56christian.heimessetmessages: + msg407049
2021-11-26 13:30:00christian.heimessetmessages: + msg407048
2021-11-26 13:10:17christian.heimessetpull_requests: + pull_request28030
2021-11-26 12:52:26christian.heimessetpull_requests: + pull_request28029
2021-11-26 12:46:25Roman Yurchaksetmessages: + msg407046
2021-11-26 12:20:20Roman Yurchaksetnosy: + Roman Yurchak
2021-11-26 09:41:56christian.heimessetdependencies: + update config.guess and config.sub, Undefinied _Py_Sigset_Converter function when HAVE_SIGSET_T not set, Cross compiling on Linux is untested, undocumented, and broken, Fix Program/_freeze_module for cross compiling Python, ctypes cfield.c defines duplicate ffi_type_* symbols
2021-11-26 08:49:24christian.heimessetpull_requests: + pull_request28025
2021-11-26 07:26:58christian.heimessetmessages: + msg407030
2021-11-25 21:51:12christian.heimessetmessages: + msg407021
2021-11-25 21:48:25christian.heimessetmessages: + msg407020
2021-11-25 20:55:56christian.heimessetpull_requests: + pull_request28021
2021-11-25 20:24:49christian.heimessetmessages: + msg407018
2021-11-25 19:29:55christian.heimessetmessages: + msg407015
2021-11-25 18:49:05brett.cannonsetmessages: + msg407014
2021-11-25 18:48:37brett.cannonsetmessages: + msg407013
2021-11-25 18:42:44brett.cannonsetmessages: + msg407012
2021-11-25 18:39:24brett.cannonsetmessages: + msg407011
2021-11-25 10:50:43christian.heimessetmessages: + msg406986
2021-11-25 10:23:16christian.heimessetnosy: + brett.cannon

messages: + msg406985
versions: + Python 3.11, - Python 3.9
2021-11-25 10:07:33christian.heimessetkeywords: + patch
nosy: + christian.heimes

pull_requests: + pull_request28008
stage: patch review
2021-05-26 22:22:08mceplsetnosy: + mcepl
2021-05-24 16:43:35Beucsetnosy: - Beuc
2021-05-21 16:14:09barrysetnosy: + barry
2021-05-20 23:49:28ethan smithsetnosy: + ethan smith
messages: + msg394081
2020-05-22 11:21:24Beucsetnosy: + Beuc
messages: + msg369598
2020-04-14 12:39:08Simon Biggssetmessages: + msg366379
2020-04-14 08:58:34pmppsetnosy: + pmpp
messages: + msg366375
2020-04-14 08:50:40xtreaksetnosy: + xtreak
2020-04-14 08:14:37serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg366374
2020-04-14 07:11:35Simon Biggscreate