classification
Title: Build of Python where make is called from subprocess, within a virtualenv, breaks on macOS
Type: compile error Stage: resolved
Components: Installation, macOS Versions: Python 3.8
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile
View: 22490
Assigned To: Nosy List: ned.deily, rdbisme, ronaldoussoren
Priority: normal Keywords:

Created on 2020-04-12 11:04 by rdbisme, last changed 2020-04-27 07:15 by ned.deily. This issue is now closed.

Files
File name Uploaded Description Edit
make.py.venv.log.gz rdbisme, 2020-04-12 11:04 make from python subprocess within a virtualenv creted with pew
make.py.venv2.log.gz rdbisme, 2020-04-12 11:05 make from python subprocess within a virtualenv created with venv
make.venv.log.gz rdbisme, 2020-04-12 11:06 make from shell within virtualenv
make.log.gz rdbisme, 2020-04-12 11:06 make from the shell without virtualenv
make.venv.log.gz rdbisme, 2020-04-12 11:07 make from shell within virtualenv
Messages (4)
msg366232 - (view) Author: Ruben (rdbisme) Date: 2020-04-12 11:04
This problem is driving me crazy. I'm working with a library (python-for-android) that builds python during a build process. 

Now, during this process, the classical workflow with `./configure` and `make` is being called from a subprocess. To try to isolate the issue, I call `make` to build Python from a file, let's call it `make.py`, in the source tree, where I simply do (after running `./configure`, manually or from a subprocess):

```
import subprocess; 
subprocess.run(["make", "-j8"])
```

I get the following error: 

```
/opt/local/bin/ranlib: file: libpython3.8.a(dynamic_annotations.o) has no symbols
/opt/local/bin/ranlib: file: libpython3.8.a(pymath.o) has no symbols
gcc     -Wl,-stack_size,1000000  -framework CoreFoundation -o python.exe Programs/python.o libpython3.8.a -ldl   -framework CoreFoundation
gcc     -Wl,-stack_size,1000000  -framework CoreFoundation -o Programs/_testembed Programs/_testembed.o libpython3.8.a -ldl   -framework CoreFoundation
./python.exe -E -S -m sysconfig --generate-posix-vars ;\
        if test $? -ne 0 ; then \
                echo "generate-posix-vars failed" ; \
                rm -f ./pybuilddir.txt ; \
                exit 1 ; \
        fi
 CC='gcc' LDSHARED='gcc -bundle -undefined dynamic_lookup    ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall'      _TCLTK_INCLUDES='' _TCLTK_LIBS=''       ./python.exe -E ./setup.py  build
```

Is this a bug of my OS, my env configuration, venv/virtualenv, Python makefile or, most probably, myself? 

I attached the log files from the following runs: 
- make from the shell without virtualenv (`make.log`)
- make from shell within virtualenv (`make.venv.log`)
- make from python subprocess without virtualenv (`make.py.log`)
- make from python subprocess within a virtualenv creted with pew (`make.py.venv.log)
- make from python subprocess within a virtualenv created with venv (`make.py.venv2.log`)

My system details:

macOS 10.15.4

Steps to reproduce: 

1. Download Python-3.8.2
2. Extract the tgz
3. Run ./configure
4. cat `make.py` in the source tree where `make.py` is:
```
import subprocess

subprocess.run(["env"])
subprocess.run(["make", "-j8"])
```
5. Run python make.py
msg366233 - (view) Author: Ruben (rdbisme) Date: 2020-04-12 11:14
Sorry, the error was missing, well, the actual error: 

```
/opt/local/bin/ranlib: file: libpython3.8.a(dynamic_annotations.o) has no symbols
/opt/local/bin/ranlib: file: libpython3.8.a(pymath.o) has no symbols
gcc     -Wl,-stack_size,1000000  -framework CoreFoundation -o python.exe Programs/python.o libpython3.8.a -ldl   -framework CoreFoundation    
gcc     -Wl,-stack_size,1000000  -framework CoreFoundation -o Programs/_testembed Programs/_testembed.o libpython3.8.a -ldl   -framework CoreFoundation    
./python.exe -E -S -m sysconfig --generate-posix-vars ;\
	if test $? -ne 0 ; then \
		echo "generate-posix-vars failed" ; \
		rm -f ./pybuilddir.txt ; \
		exit 1 ; \
	fi
 CC='gcc' LDSHARED='gcc -bundle -undefined dynamic_lookup    ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall' 	_TCLTK_INCLUDES='' _TCLTK_LIBS='' 	./python.exe -E ./setup.py  build
running build
running build_ext
error: [Errno 2] No such file or directory: '/usr/local/include/python3.8/pyconfig.h'
```
msg366281 - (view) Author: Ruben (rdbisme) Date: 2020-04-13 01:53
I just tested on a fresh macOS 10.15.3 Virtual Machine (on the same host machine as the one of previous messages) and I can reproduce the problem: 

1. Install Command Line Tools (xcode-build --install)
2. Download Python 3.8.1
3. Extract tarball
4. Create a file make.py with the following python code
```
import subprocess

subprocess.run(["./configure"])
subprocess.run(["make", "-j4"])
```
and save it in the Python-3.8.1 source tree
6. Create a venv doing `python -m venv venv` in the source tree
7. Activate the virtual env `source venv/bin/activate`
5. Run it doing `python3 make.py`

The build fails with the error described before. 

If you now, instead, run the make directly `make -j8` the buids proceeds as expected.
msg367404 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-04-27 07:15
Thanks for the easy-to-reproduce test case! It looks like you are running into the problem described in Issue22490 where venv with macOS framework builds could run into problems. A fix for this issue has recently been merged and will be released in upcoming 3.8.3 (a 3.8.3 release candidate should be available in the very near future) and 3.7.8.  I've verified that your test case now works with the current HEAD of 3.8 and fails with 3.8.2.
History
Date User Action Args
2020-04-27 07:15:57ned.deilysetstatus: open -> closed
superseder: Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile
messages: + msg367404

resolution: duplicate
stage: resolved
2020-04-13 01:53:25rdbismesetmessages: + msg366281
2020-04-12 11:14:44rdbismesetmessages: + msg366233
2020-04-12 11:07:32rdbismesetfiles: + make.venv.log.gz
2020-04-12 11:06:57rdbismesetfiles: + make.log.gz
2020-04-12 11:06:32rdbismesetfiles: + make.venv.log.gz
2020-04-12 11:05:39rdbismesetfiles: + make.py.venv2.log.gz
2020-04-12 11:04:21rdbismecreate