classification
Title: lzma.h file not found building on macOS
Type: Stage: resolved
Components: macOS Versions:
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: jaraco, ned.deily, ronaldoussoren
Priority: normal Keywords:

Created on 2020-06-02 00:28 by jaraco, last changed 2020-11-26 23:51 by jaraco. This issue is now closed.

Messages (7)
msg370582 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-06-02 00:28
Attempting to build Python on macOS following [the instructions](https://devguide.python.org/setup/#macos-and-os-x) (for Homebrew).

xz is installed:

```
$ brew --prefix xz                                                                                                            
/Users/jaraco/.local/homebrew/opt/xz
```

Yet, after running `./configure`, which makes no mention of "xz" or "lzma", "make" fails with:

```
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I./Include -I. -I/Users/jaraco/code/public/cpython/Include -I/Users/jaraco/code/public/cpython -c /Users/jaraco/code/public/cpython/Modules/_lzmamodule.c -o build/temp.macosx-10.15-x86_64-3.9/Users/jaraco/code/public/cpython/Modules/_lzmamodule.o
/Users/jaraco/code/public/cpython/Modules/_lzmamodule.c:16:10: fatal error: 'lzma.h' file not found
#include <lzma.h>
         ^~~~~~~~
1 error generated.
```

Yet the file is there: 

```
$ ls ~/.local/homebrew/opt/xz/include                                                                                         
lzma   lzma.h
```

What's missing from the instructions that I should be doing?
msg370583 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-06-02 00:31
This issue doesn't happen on my mac that installs homebrew globally. Do the build instructions assume Homebrew is installed system-wide?
msg370585 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-06-02 00:59
In [this commit](https://github.com/jaraco/jaraco.develop/commit/e3e5f66ca6693d8ec3abd31d99d491f6dfa1f67d), I include "xz" in the routine. Should Python's developer docs include something like that to ensure compatibility?
msg370590 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-06-02 02:07
For Homebrew users, the Devguide currently recommends:

$ brew install openssl xz gdbm

./configure --with-pydebug --with-openssl=$(brew --prefix openssl)

I don't use Homebrew myself so I may be missing something
obvious but, after a bit of experimentation,
I think the following might be the simplest version of the
above that works with Homebrew installed in a non-default location:

$ CPPFLAGS="-I$(brew --prefix)/include" \
  LDFLAGS="-L$(brew --prefix)/lib" \
  ./configure --with-pydebug \
  --with-openssl=$(brew --prefix openssl)

Could you try it on your systems, please?

BTW, I notice that you are also installing zlib. There was a
longstanding bug in the 2.7 setup.py that caused the system
zlib to not be found in some cases. That was fixed some time
back. So there is no normally no need to install a third-party
zlib; current Python builds will automatically use the
Apple-supplied system zlib.

Since, like openssl, Homebrew installs zlib "keg-only",
you would need to explicitly reference its locations:

$ CPPFLAGS="-I$(brew --prefix)/include -I$(brew --prefix zlib)/include" \
  LDFLAGS="-L$(brew --prefix)/lib -L$(brew --prefix zlib)/lib" \
  ./configure --with-pydebug \
  --with-openssl=$(brew --prefix openssl)

But, at the moment, both the 10.15 system zlib and the Homebrew
zlib are the same version, 1.2.11, so it doesn't seem necessary.
msg370691 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-06-04 01:03
In [this latest routine](https://github.com/jaraco/jaraco.develop/blob/6469c7a61e7349b93f191df38eed6cd020dd79be/jaraco/develop/macos-build-python.py), on my macOS workstation with Homebrew installed locally, Python builds successfully with just a few outliers:

```
Python build finished successfully!
The necessary bits to build these optional modules were not found:
_gdbm                 ossaudiodev           spwd               
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd                
time                                                           
```

The output is the same as on the system with Homebrew system-installed, except for `_gdbm`, which seems only to work on system-installed Homebrew.

That's much cleaner and addresses the reported issue. I guess I should have been more careful to execute exactly what the dev docs said rather than my routine which must have accumulated some cruft as you recognized.

Thanks for the help.

If you have any other hints as to how to ensure _gdbm builds, I'm open to suggestions. Thanks.
msg381914 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-11-26 20:31
Is this issue resolved?
msg381923 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-11-26 23:51
Yes. If I trust my message from earlier, this issue is resolved. Closing now.
History
Date User Action Args
2020-11-26 23:51:53jaracosetstatus: open -> closed
resolution: not a bug
messages: + msg381923

stage: resolved
2020-11-26 20:31:57ronaldoussorensetmessages: + msg381914
2020-06-04 01:03:21jaracosetmessages: + msg370691
2020-06-02 02:07:30ned.deilysetmessages: + msg370590
2020-06-02 00:59:10jaracosetmessages: + msg370585
2020-06-02 00:31:01jaracosetmessages: + msg370583
2020-06-02 00:28:35jaracosetnosy: + ronaldoussoren, ned.deily
components: + macOS
2020-06-02 00:28:11jaracocreate