This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Difficulties building tip of tree Python for emscripten
Type: Stage: resolved
Components: Versions:
process
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, hoodchatham, pmpp
Priority: normal Keywords:

Created on 2022-04-05 19:27 by hoodchatham, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
Makefile hoodchatham, 2022-04-05 21:29
Messages (8)
msg416807 - (view) Author: Hood Chatham (hoodchatham) * Date: 2022-04-05 19:27
I am trying to build tot Python for Emscripten to test recent changes and having trouble. The problem is caused by recent changes since 3.11.0a6 to `sre`. I build a build Python 3.11 from tot, and provided it to `./configure --with-build-python=$(HOSTBUILDDIR)/python`. However, I still see invocations to the system Python 3.11 at `/usr/local/bin/python3.11`.

The problem seems to be that the generated `Makefile` has a bunch of stuff collapsed into one line:
```
PYTHON_FOR_BUILD=_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) /src/cpython/build/Python-3.11.0dev0/builddir/build-host/python
```
manually fixing this to:
```
PYTHON_FOR_BUILD=/src/cpython/build/Python-3.11.0dev0/builddir/build-host/python
_PYTHON_PROJECT_BASE=$(abs_builddir) 
_PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) 
PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib 
_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH)
```
fixes the problem.

My configure invocation is as follows:
```
	( \
		cd $(BUILDDIR); \
		CONFIG_SITE=./config.site-wasm32-emscripten emconfigure \
		  ../../configure \
		  	  --cache-file=$(ROOT)/python.config.cache \
			  CFLAGS="${PYTHON_CFLAGS}" \
			  CPPFLAGS="-I$(SQLITEBUILD) -I$(BZIP2BUILD) -I$(ZLIBBUILD)" \
			  PLATFORM_TRIPLET="$(PLATFORM_TRIPLET)" \
			  --with-build-python=$(HOSTBUILDDIR)/python \
			  --without-pymalloc \
			  --disable-shared \
			  --disable-ipv6 \
			  --enable-big-digits=30 \
			  --enable-optimizations \
			  --host=wasm32-unknown-emscripten\
			  --build=$(shell $(EXTRACTDIR)/config.guess) \
			  --prefix=$(INSTALL)  \
	)
```
msg416812 - (view) Author: Hood Chatham (hoodchatham) * Date: 2022-04-05 19:58
Okay I found the problem. Elsewhere in my Makefiles `PYTHON_FOR_BUILD` was getting overwritten.
msg416815 - (view) Author: Hood Chatham (hoodchatham) * Date: 2022-04-05 20:28
I take it back, I am still having this problem.
msg416816 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2022-04-05 20:39
Please attach config.log and Makefile from the builddir.
msg416819 - (view) Author: Hood Chatham (hoodchatham) * Date: 2022-04-05 21:29
config.log is apparently 1.7 MB and when I try to upload I get "Error 413: Entity Too Large". I've attached the Makefile.
msg416820 - (view) Author: Hood Chatham (hoodchatham) * Date: 2022-04-05 21:52
There is still a pretty good change I am making some dumb mistake.
msg416821 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2022-04-05 21:57
You are building a cross-build with "--with-build-python=/usr/local/bin/python3.11". Is the interpreter up to date?

During alpha and beta phase, the build Python interpreter should be built from the exact same git commit as the sources in "../..". If the interpreter deviates from the sources in your srcdir, then Python byte code (.pyc), frozen byte code, and re engine can have mismatching magic.

I recommend that you build inside an environment that does not have Python 3.11 installed. Instead build a build Python interpreter from the same checkout that you later use for cross-compilation. "./configure && make" is sufficient. You don't have to install the build interpreter. "--with-build-python=$(pwd)/../build/python" works fine.

We use this approach in python-wasm:

mkdir -p cpython/builddir/build
pushd cpython/builddir/build
../../configure -C
make -j$(nproc)
popd

mkdir -p cpython/builddir/emscripten-browser
pushd cpython/builddir/emscripten-browser
CONFIG_SITE=../../Tools/wasm/config.site-wasm32-emscripten \
  emconfigure ../../configure -C \
    --host=wasm32-unknown-emscripten \
    --build=$(../../config.guess) \
    --with-build-python=$(pwd)/../build/python
emmake make -j$(nproc)


It's going to get easier and less painful as soon as we reach beta phase. During beta the byte code will only change when it is required for a bug fix.
msg416824 - (view) Author: Hood Chatham (hoodchatham) * Date: 2022-04-05 22:33
Okay I got it working. I was making several mistakes that each caused the same symptoms so it was hard to track them all down.

Thanks again for your help!
History
Date User Action Args
2022-04-11 14:59:58adminsetgithub: 91388
2022-04-05 22:33:35hoodchathamsetstatus: open -> closed

messages: + msg416824
2022-04-05 21:57:31christian.heimessetmessages: + msg416821
2022-04-05 21:52:35hoodchathamsetmessages: + msg416820
2022-04-05 21:29:20hoodchathamsetfiles: + Makefile

messages: + msg416819
2022-04-05 20:39:10christian.heimessetmessages: + msg416816
2022-04-05 20:28:42hoodchathamsetstatus: closed -> open

messages: + msg416815
2022-04-05 19:58:40hoodchathamsetstatus: open -> closed

messages: + msg416812
stage: resolved
2022-04-05 19:41:49pmppsetnosy: + pmpp
2022-04-05 19:27:14hoodchathamcreate