For macOS, if I understand correctly what you want to do, the way to avoid using "install_name_tool" to modify the id name of the libpython* dynamic shared library is to pass the desired name to the link step when building the shared library using ld's "-install_name" option.  For macOS, the Makefile rule that builds the shared lib is the one labeled libpython$(LDVERSION).dylib. If you also need to build a working python executable that uses that shared library, you will want to pass the proper rpath command into the link step of python itself which happens in the $(BUILDPYTHON) Makefile rule.

On macOS there are three main variants when building core python: 1. static library, 2. shared library, 3. framework (macOS-only).  The most common variant for a full Python installation is a framework build.  When building bits intended for embedding, in theory either the "shared library" or the "framework" variants should work but, to keep things simpler, let's just stick to "shared library".  If so, it looks like it is possible to configure the build properly by just using existing configuration variables.  The keys are:

- use LDFLAGS_NODIST to pass in the library install name (note that the option here will also get passed to the link step for the python executable but the value is ignored if the output of the link is not a shared library)

- use LINKFORSHARED to pass in the desired runtime rpath value to the build of the executable.

Putting it all together, a simplified build might be:

LDFLAGS_NODIST='-Wl,-install_name,@rpath/libpython3.8.dylib' \
LINKFORSHARED="-framework CoreFoundation -Wl,-rpath,@loader_path/../lib" \
./configure --enable-shared --prefix=/path/to/installed/location

make -j3

make install

Note "-framework CoreFoundation" is a default value for "LINKFORSHARED" on macOS and should be included in the overridden value.

You can verify that the expected shared libraries are being used at run time by enabling some debugging info in the dynamic loader:

DYLD_PRINT_LIBRARIES= /path/to/installed/location/python3

Obviously, this is a simplified configuration. The macOS dynamic loader has a number of options for specifying library paths; some of the options differ from those available in other unix-y systems.  The following macOS man pages have more details: dyld (in particular, the DYNAMIC LIBRARY LOADING section which explains @executable_path, @loader_path, and @rpath options), install_name_tool, ld, cc.

Let me know if the above is what you were looking for.


P.S. One gotcha that may show up during development building and testing with "--enable-shared" is that when trying to run the interpreter from the build directory you will need to use runtime env variables to give the path to the (uninstalled) shared python library.  If you don't, you will either get a segfault or, worse, the newly-built executable may run with a previously installed library.

$ ./python # or ./python.exe depending on file system type
dyld: Library not loaded: /tmp/l/lib/libpython3.8.dylib
  Referenced from: /Users/nad/Projects/PyDev/active/dev/3x/source/./python
  Reason: image not found
Abort trap: 6
$ DYLD_LIBRARY_PATH=$(pwd) ./python
Python 3.8.0a4+ (heads/master:6de4574c63, May 24 2019, 19:45:27)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin [...]

That's one reason I recommend avoiding --enable-shared when possible, especially on macOS.