classification
Title: "make -j9 install" fails because bininstall target requires the libainstall target
Type: Stage: patch review
Components: Build Versions: Python 3.6, Python 3.4, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, doko, martin.panter, python-dev, vstinner
Priority: normal Keywords: patch

Created on 2015-11-22 14:31 by vstinner, last changed 2015-12-13 20:27 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
bininstall.patch vstinner, 2015-11-22 14:31
bininstall-2.patch vstinner, 2015-11-25 14:31 review
bininstall-3.patch vstinner, 2015-11-27 15:18 review
bininstall-4.patch martin.panter, 2015-11-28 02:30 Separate $(LIBPC) rule review
Messages (12)
msg255102 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-11-22 14:31
I'm unable to install Python 3.6, "make install" doesn't copy .py files because the bininstall target failed. It failed because /opt/py36/lib/pkgconfig doesn't exist.

Try:
---
export MAKEFLAGS=-j9
rm -rf /opt/py36
./configure --prefix=/opt/py36
make
make install   # you may need 'sudo make install'
---

Attached patch fixes the Makefile, but I don't know if my change is correct.
msg255103 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-11-22 14:33
Output:
---
$ make install --debug=v
(...)
  Must remake target 'bininstall'.
(...)
        (cd /opt/py36/lib/pkgconfig; ln -s python-3.6.pc python-3.6dm.pc); (...)
(...)
/bin/sh: ligne 4 : cd: /opt/py36/lib/pkgconfig: Aucun fichier ou dossier de ce type
---
msg255108 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2015-11-22 17:18
This problem in bininstall target seems to exist since Python 2.7 and 3.1.

bininstall target depends on altbininstall target, which contains:

        @for i in $(BINDIR) $(LIBDIR); \
        do \
                if test ! -d $(DESTDIR)$$i; then \
                        echo "Creating directory $$i"; \
                        $(INSTALL) -d -m $(DIRMODE) $(DESTDIR)$$i; \
                else    true; \
                fi; \
        done

For consistency with other things, I would suggest to add the following at the beginning of bininstall target (instead of making bininstall target depend on libainstall target):

        @for i in $(LIBPC); \
        do \
                if test ! -d $(DESTDIR)$$i; then \
                        echo "Creating directory $$i"; \
                        $(INSTALL) -d -m $(DIRMODE) $(DESTDIR)$$i; \
                else    true; \
                fi; \
        done
msg255359 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-11-25 14:31
To be clear: the bug doesn't occur when Python is not installed "in parallel" (without MAKEFLAGS=-j9).

"For consistency with other things, I would suggest to add the following at the beginning of bininstall target (instead of making bininstall target depend on libainstall target):"

Ok. Does my new patch look better?

Should I apply my fix to Python 2.7, 3.4, 3.5 and default (3.6)?
msg255393 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-11-25 23:58
$(LIBPC) is a single directory name, so I suggest dropping the “for” loop.

Patch 2 should avoid the practical race condition. But there is technically still a race with “libainstall” and “bininstall” both testing and creating the same directory. Maybe we should factor it out, something like:

$(DESTDIR)$(LIBPC):
	@echo "Creating directory $@"
	@$(INSTALL) -d -m $(DIRMODE) $@
. . .
bininstall: altbininstall $(DESTDIR)$(LIBPC)
. . .
libainstall: all python-config $(DESTDIR)$(LIBPC)
	@for i in $(LIBDIR) $(LIBPL); \
	. . .

Looking at the history, the test-then-install code comes from 20 years ago: <https://hg.python.org/cpython/rev/d41d89eb43e5#l1.150>. I would say calling install unconditionally (without a test or echo message) might be simpler, but that’s probably getting out of scope here.
msg255462 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-11-27 15:18
> $(LIBPC) is a single directory name, so I suggest dropping the “for” loop.

Ok, here is a version without loop. I also added "-" before the if to mimick other lines of the bininstall target.

Does it look better now?

> (...) there is technically still a race with “libainstall” and “bininstall” both testing and creating the same directory.

Sorry, I don't understand. Is it an issue in practice to create the same directory twice "at the same time"? Are you able to reproduce the bug?

Please try my use case: "./configure && make && make -j10 install" and enjoy the partial installation :-) (The python binary is installed in bin/ but it doesn't work because the stdlib is completly missing.)
msg255519 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-11-28 02:29
I agree your patches should fix your practical bug (I can reproduce it, two out of three times). My concern is more about making the code cleaner and less likely to grow problems in the future. :) The worse case with patch 3 would be the “Creating directory” message being echoed twice. I am just trying to get the makefile cleaner.

If that block compiled a file, instead of just creating a directory, then it would be a worse problem. The second compile command could truncate the first version, causing errors later on.

Also, I don’t believe adding the dash (-) prefix is a good idea here. It tells Make to keep going even if the command failed. In this case, if the directory already exists, the “if” statement will succeed, and the only way it can fail is if $(INSTALL) fails, which I think we would want to know about. The original “for” loop under libainstall does not include a dash.

bininstall-4.patch implements my suggestion from before.
msg255575 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-11-29 14:37
bininstall-4.patch: IMHO it's overkill and makes Makefile more complex to follow. My patch  bininstall.patch proposed something similar, but Arfrever asked me to modify bininstall target. IMHO Getting "Creating directory xxx" message twice is a minor issue.

It becomes annoying to have nitpicking on my patch, I hate autotools, I hate having to care of Makefile, etc. Maybe I should just abandon my change, it looks like nobody cares that Python cannot be installed with -j9 anyway :-)
msg255599 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-11-29 22:34
Please go ahead with bininstall-3.patch if you prefer. It certainly avoids the problem you reported. Although I still encourage you to not add the dash before the command.
msg256334 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-12-13 20:22
New changeset c03ef448b5b2 by Victor Stinner in branch '2.7':
Issue #25696: Fix installation of Python on UNIX with make -j9.
https://hg.python.org/cpython/rev/c03ef448b5b2

New changeset 87d96b349ff5 by Victor Stinner in branch '3.5':
Issue #25696: Fix installation of Python on UNIX with make -j9.
https://hg.python.org/cpython/rev/87d96b349ff5

New changeset 8dd594d36704 by Victor Stinner in branch 'default':
(Merge 3.5) Issue #25696: Fix installation of Python on UNIX with make -j9.
https://hg.python.org/cpython/rev/8dd594d36704
msg256335 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-12-13 20:27
New changeset 94910d210ef4 by Victor Stinner in branch '2.7':
Issue #25696: Don't ignore errors in 'make bininstall' on creating $(LIBPC) directory
https://hg.python.org/cpython/rev/94910d210ef4

New changeset d28268c47421 by Victor Stinner in branch '3.5':
Issue #25696: Don't ignore errors in 'make bininstall' on creating $(LIBPC) directory
https://hg.python.org/cpython/rev/d28268c47421
msg256336 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-12-13 20:27
> Please go ahead with bininstall-3.patch if you prefer.

Ok done.

> It certainly avoids the problem you reported. Although I still encourage you to not add the dash before the command.

Oops, fixed.
History
Date User Action Args
2015-12-13 20:27:29vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg256336
2015-12-13 20:27:07python-devsetmessages: + msg256335
2015-12-13 20:22:40python-devsetnosy: + python-dev
messages: + msg256334
2015-11-29 22:34:09martin.pantersetmessages: + msg255599
2015-11-29 14:37:06vstinnersetmessages: + msg255575
2015-11-28 02:30:05martin.pantersetfiles: + bininstall-4.patch
2015-11-28 02:29:17martin.pantersetmessages: + msg255519
2015-11-27 15:18:46vstinnersetfiles: + bininstall-3.patch

messages: + msg255462
2015-11-25 23:58:23martin.pantersetnosy: + martin.panter

messages: + msg255393
stage: patch review
2015-11-25 14:31:54vstinnersettitle: "MAKEFLAGS=-j9 make install" fails because bininstall target requires the libainstall target -> "make -j9 install" fails because bininstall target requires the libainstall target
2015-11-25 14:31:10vstinnersetfiles: + bininstall-2.patch

messages: + msg255359
title: "make install" fails because bininstall target requires the libainstall target -> "MAKEFLAGS=-j9 make install" fails because bininstall target requires the libainstall target
2015-11-22 17:18:12Arfreversetmessages: + msg255108
versions: + Python 2.7, Python 3.4, Python 3.5
2015-11-22 14:33:59vstinnersetmessages: + msg255103
2015-11-22 14:31:42vstinnercreate