when trying to build rc1 from a b4 venv to prepare a python host for cross compilation on ubuntu xenial flavour x64

i got :


cd /tmp
python3.7 -m venv testenv
cd testenv/
. bin/activate
(testenv) /tmp/testenv $ tar xf /tmp/Python-3.8.0rc1.tar.xz && cd Python-3.8.0rc1

CC=clang ./configure --prefix=/tmp/python3.host --with-system-ffi --disable-ipv6 --without-ensurepip --with-c-locale-coercion --disable-shared && make

./python -E ./setup.py  build
step will fail for the reason i suspect is srcdir is taken from system python3.8 if installed ( /usr/local in that case )

$ ./python -E 
Python 3.8.0rc1 (default, Oct  7 2019, 15:16:07) 
[Clang 6.0.0 (tags/RELEASE_600/final)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_var('srcdir')
'/usr/local/lib/python3.8/config-3.8-x86_64-linux-gnu'
>>>

It's not a Python 3.8 regression. Python 3.7 has the same behavior.


cd $HOME
python3.7 -m venv testenv
cd testenv/
. bin/activate

wget https://www.python.org/ftp/python/3.8.0/Python-3.8.0rc1.tar.xz
tar -xf Python-3.8.0rc1.tar.xz && cd Python-3.8.0rc1
CC=clang ./configure --prefix=/tmp/python3.host --with-system-ffi --disable-ipv6 --without-ensurepip --with-c-locale-coercion --disable-shared && make

Python 3.8 error:

"""
$ make
./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
	echo "generate-posix-vars failed" ; \
	rm -f ./pybuilddir.txt ; \
	exit 1 ; \
fi
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = './python'
  isolated = 0
  environment = 0
  user site = 1
  import site = 0
  sys._base_executable = '/home/vstinner/testenv/Python-3.8.0rc1/python'
  sys.base_prefix = '/tmp/python3.host'
  sys.base_exec_prefix = '/tmp/python3.host'
  sys.executable = '/home/vstinner/testenv/Python-3.8.0rc1/python'
  sys.prefix = '/tmp/python3.host'
  sys.exec_prefix = '/tmp/python3.host'
  sys.path = [
    '/tmp/python3.host/lib/python38.zip',
    '/tmp/python3.host/lib/python3.8',
    '/tmp/python3.host/lib/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007ff6a75f6740 (most recent call first):
<no Python frame>
generate-posix-vars failed
make: *** [Makefile:592: pybuilddir.txt] Error 1
"""


cd ..
wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tar.xz
tar -xf Python-3.7.4.tar.xz 
cd Python-3.7.4/
CC=clang ./configure --prefix=/tmp/python3.host --with-system-ffi --disable-ipv6 --without-ensurepip --with-c-locale-coercion --disable-shared

Python 3.7 error:

"""
$ make
./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
	echo "generate-posix-vars failed" ; \
	rm -f ./pybuilddir.txt ; \
	exit 1 ; \
fi
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: initfsencoding: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f5a77f5a740 (most recent call first):
/bin/sh : ligne 5 : 19592 Aborted                 (core dumped)./python -E -S -m sysconfig --generate-posix-vars
generate-posix-vars failed
make: *** [Makefile:606: pybuilddir.txt] Error 1
"""

The problem is that Python looks for pyvenv.cfg in the program directory or in its parent directory. For example:

/tmp/testenv/Python-3.8.0rc1/python looks for /tmp/testenv/Python-3.8.0rc1/pyvenv.cfg (doesn't exist) and 
/tmp/testenv/Python-3.8.0rc1/pyvenv.cfg (exists). On my system, pyvenv.cfg contains:
---
home = /usr/bin
include-system-site-packages = false
version = 3.7.4
---

We should add an option to ignore pyvenv.cfg, or modify an existing option like -I to ignore pyvenv.cfg.