classification
Title: test_site fails in macOS-PR VSTS builds for 3.7 branch
Type: Stage: resolved
Components: macOS, Tests Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: inada.naoki, ned.deily, pablogsal, ronaldoussoren, vstinner, xtreak
Priority: normal Keywords: patch

Created on 2018-07-21 07:32 by xtreak, last changed 2018-08-10 07:17 by inada.naoki. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 8430 merged inada.naoki, 2018-07-24 05:42
PR 8433 closed inada.naoki, 2018-07-24 06:35
PR 8491 merged miss-islington, 2018-07-27 07:55
PR 8493 merged inada.naoki, 2018-07-27 08:07
Messages (10)
msg322082 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2018-07-21 07:32
Mac OS builds fail due to test_site in VSTS for 3.7 branches only

Ref : https://python.visualstudio.com/cpython/_build/results?buildId=17921&view=logs

2018-07-21T07:01:08.5912930Z Traceback (most recent call last):
2018-07-21T07:01:08.5927520Z   File "/Users/vsts/agent/2.136.1/work/1/s/Lib/test/test_site.py", line 199, in test_s_option
2018-07-21T07:01:08.5942120Z     self.assertIn(usersite, sys.path)
2018-07-21T07:01:08.5959870Z AssertionError: '/Users/vsts/.local/lib/python3.7/site-packages' not found in ['/Users/vsts/agent/2.136.1/work/1/s', '/usr/local/lib/python37.zip', '/Users/vsts/agent/2.136.1/work/1/s/Lib', '/Users/vsts/agent/2.136.1/work/1/s/build/lib.macosx-10.13-x86_64-3.7-pydebug', '/usr/local/lib/python3.7/site-packages']
2018-07-21T07:01:08.5971110Z 
2018-07-21T07:01:08.5985860Z ======================================================================
2018-07-21T07:01:08.6000880Z FAIL: test_startup_imports (test.test_site.StartupImportTests)
2018-07-21T07:01:08.6017050Z ----------------------------------------------------------------------
2018-07-21T07:01:08.6033630Z Traceback (most recent call last):
2018-07-21T07:01:08.6048270Z   File "/Users/vsts/agent/2.136.1/work/1/s/Lib/test/test_site.py", line 487, in test_startup_imports
2018-07-21T07:01:08.6063180Z     self.assertFalse(modules.intersection(collection_mods), stderr)
2018-07-21T07:01:08.6079990Z AssertionError: {'reprlib', 'heapq', 'collections', 'keyword', 'operator', 'functools', 'types'} is not false : import _frozen_importlib # frozen

I couldn't find any related issues in search and hence opened this. Feel free to close this if it's a duplicate or fixed in some other branch.

Thanks
msg322083 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2018-07-21 07:44
It seems that the last successful build in 3.7 was for https://github.com/python/cpython/commit/339e0c1296c61c9dbc1f8f880c5c668bf4007e5e and all PRs after it had the same failure.

Sorry for not adding the relevant components during ticket creation.

Thanks
msg322207 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2018-07-23 12:40
Adding @vstinner as a friendly reminder since buildbot related failures are notified. This issue causes all the PRs for 3.7 on Mac OS VSTS builds to fail since July 19 though other Mac related builds are passing.

Feel free to remove yourself or close this if it's irrelevant.

Thanks
msg322283 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2018-07-24 09:33
Maybe, vsts machine installs Homebrew's Python.
Homebrew's Python installes /usr/local/lib/python3.7/sitecustomize.py

https://github.com/Homebrew/homebrew-core/blob/0ce9a5a481b3472608fe4e4188cb5d243deebb99/Formula/python.rb#L295-L329

When running tests, it is imported.
It causes test_startup_imports fail.

Maybe, we can use random prefix instead of default /usr/local to avoid this conflict.
msg322473 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2018-07-27 09:03
New changeset 777cdd94b9bebd5b22df0fc293aa078d5537b988 by INADA Naoki in branch '3.6':
bpo-34177: vsts: Avoid conflict with Homebrew Python (GH-8430)
https://github.com/python/cpython/commit/777cdd94b9bebd5b22df0fc293aa078d5537b988
msg322508 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-07-27 16:35
Thank you for the fix Naokia!
msg322936 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-08-02 10:06
This error is happening in the x86-64 High Sierra 3.7 buildbot:

https://buildbot.python.org/all/#/builders/147/builds/174

======================================================================
FAIL: test_startup_imports (test.test_site.StartupImportTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.7.billenstein-sierra/build/Lib/test/test_site.py", line 487, in test_startup_imports
    self.assertFalse(modules.intersection(collection_mods), stderr)
AssertionError: {'collections', 'heapq', 'types', 'functools', 'keyword', 'reprlib', 'operator'} is not false : import _frozen_importlib # frozen
msg322940 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2018-08-02 10:15
Same fix can be applied to buildbot.

This is caused by (a) Homebrew installs Python to /usr/local with (hacky) sitecustomize, and (b) Python's default prefix is /usr/local.

I think test should be run without conflicting with another installation.
msg322941 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-02 10:37
>  Same fix can be applied to buildbot.

Sure. Go ahead :-) The buildbot configuration can be found at:
https://github.com/python/buildmaster-config/

Send a pull request, and Zachary Ware (or me) will  review it. Zach is the expert here ;-)
msg323274 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2018-08-08 11:15
I think this can be closed with https://github.com/python/buildmaster-config/commit/9a456462fb891328b5b8a170522d5f56f480fdfb. The buildbots are also green now with the failure reported in https://buildbot.python.org/all/#/builders/147/builds/174 fixed with the commit.

Thanks
History
Date User Action Args
2018-08-10 07:17:02inada.naokisetstatus: open -> closed
resolution: fixed
2018-08-08 11:15:24xtreaksetmessages: + msg323274
2018-08-02 10:37:28vstinnersetresolution: fixed -> (no value)
messages: + msg322941
2018-08-02 10:15:21inada.naokisetmessages: + msg322940
2018-08-02 10:06:56pablogsalsetstatus: closed -> open
2018-08-02 10:06:12pablogsalsetnosy: + pablogsal
messages: + msg322936
2018-07-27 16:35:42vstinnersetmessages: + msg322508
2018-07-27 09:04:15inada.naokisetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-07-27 09:03:58inada.naokisetmessages: + msg322473
2018-07-27 08:07:30inada.naokisetpull_requests: + pull_request8014
2018-07-27 07:55:58miss-islingtonsetpull_requests: + pull_request8013
2018-07-24 09:33:33inada.naokisetnosy: + inada.naoki
messages: + msg322283
2018-07-24 06:35:17inada.naokisetpull_requests: + pull_request7958
2018-07-24 05:42:34inada.naokisetkeywords: + patch
stage: patch review
pull_requests: + pull_request7956
2018-07-23 13:54:11vstinnersetcomponents: + Tests, - Build
2018-07-23 13:54:04vstinnersetnosy: + ronaldoussoren, ned.deily
components: + macOS
2018-07-23 12:40:41xtreaksetnosy: + vstinner
messages: + msg322207
2018-07-21 07:44:49xtreaksetmessages: + msg322083
components: + Build
versions: + Python 3.7
2018-07-21 07:32:04xtreakcreate