classification
Title: test_site failure when .local/lib/pythonX.Y/site-packages hasn't been created yet
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.6, Python 3.5, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: berker.peksag, haypo, martin.panter, merwok, ned.deily, pitrou, serhiy.storchaka, tarek, yjchen
Priority: normal Keywords:

Created on 2009-11-08 00:27 by pitrou, last changed 2017-05-02 09:45 by haypo. This issue is now closed.

Messages (11)
msg95031 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-11-08 00:27
On that buildbot, /var/lib/buildbot is a symlink to /home/buildbot and I
get the following failure:

======================================================================
FAIL: test_s_option (test.test_site.HelperFunctionsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/home/buildbot/cpython-ucs2/3.x.pitrou-ubuntu/build/Lib/test/test_site.py",
line 107, in test_s_option
    self.assertIn(usersite, sys.path)
AssertionError: '/var/lib/buildbot/.local/lib/python3.2/site-packages'
not found in ['/usr/local/lib/python32.zip',
'/home/buildbot/cpython-ucs2/3.x.pitrou-ubuntu/build/Lib',
'/home/buildbot/cpython-ucs2/3.x.pitrou-ubuntu/build/Lib/plat-linux2',
'/home/buildbot/cpython-ucs2/3.x.pitrou-ubuntu/build/Modules',
'/home/buildbot/cpython-ucs2/3.x.pitrou-ubuntu/build/build/lib.linux-x86_64-3.2-pydebug']

----------------------------------------------------------------------
msg95032 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-11-08 00:28
Well, I'm not sure the symlink is actually the problem. Apparently the
".local" directory hadn't been created when the first test run.
msg122093 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2010-11-22 04:45
Is this still relevant?
msg138003 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2011-06-09 16:21
Related: I found that test_site tries to create the user site-packages dir if it does not exist, but it does not remove it after the test run.
msg221301 - (view) Author: YJ Chen (yjchen) Date: 2014-06-22 19:46
Works for 3.4 and 3.5 but could not successfully run test on 2.7.

Error:
======================================================================
FAIL: test_getsitepackages (test.test_site.HelperFunctionsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/7.1/lib/python2.7/test/test_site.py", line 247, in test_getsitepackages
    self.assertEqual(len(dirs), 4)
AssertionError: 2 != 4

======================================================================
FAIL: test_getuserbase (test.test_site.HelperFunctionsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/7.1/lib/python2.7/test/test_site.py", line 210, in test_getuserbase
    site.getuserbase())
AssertionError: /Users/yjchen/Library/Python/2.7

======================================================================
FAIL: test_s_option (test.test_site.HelperFunctionsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/7.1/lib/python2.7/test/test_site.py", line 193, in test_s_option
    self.assertEqual(rc, 1)
AssertionError: 0 != 1

----------------------------------------------------------------------
msg221320 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-06-22 22:23
YJ, the test_site failures you are seeing should have been fixed by the changes for Issue10881 (82c4f094f811) that were released with Python 2.7.3.  Please update your Python 2.7 to the latest release (currently 2.7.7).

Otherwise, unless somebody is able to reproduce the original failure (I can't so far), I think we should close this issue.
msg225546 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-08-19 22:37
Closing, since this problem apparently no longer occurs
msg256289 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-12-12 17:41
This problem still occurs.

http://buildbot.python.org/all/builders/x86-64%20Ubuntu%2014.04%20ICC%20Debug%203.5/builds/275/steps/test/logs/stdio

======================================================================
FAIL: test_s_option (test.test_site.HelperFunctionsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/buildarea/3.5.intel-ubuntu-icc/build/Lib/test/test_site.py", line 169, in test_s_option
    self.assertIn(usersite, sys.path)
AssertionError: '/var/lib/buildbot/.local/lib/python3.5/site-packages' not found in ['', '/usr/local/lib/python35.zip', '/buildarea/3.5.intel-ubuntu-icc/build/Lib', '/buildarea/3.5.intel-ubuntu-icc/build/Lib/plat-linux', '/buildarea/3.5.intel-ubuntu-icc/build/build/lib.linux-x86_64-3.5-pydebug']

----------------------------------------------------------------------

On 3.5 the test is failed randomly, on 3.4 it is failed all time.

I can't reproduce this locally. If remove ~/.local/lib/pythonX.Y and run test_site, it creates the directory and modifies sys.path (this is a bug).

$ rm -rf ~/.local/lib/python3.4
$ ./python -m test.regrtest -uall -vv test_site
...
Ran 21 tests in 0.277s

OK (skipped=2)
Warning -- sys.path was modified by test_site
  Before: (3071378764, ['', '/usr/local/lib/python34.zip', '/home/serhiy/py/cpython3.4/Lib', '/home/serhiy/py/cpython3.4/Lib/plat-linux', '/home/serhiy/py/cpython3.4/build/lib.linux-i686-3.4'], ['', '/usr/local/lib/python34.zip', '/home/serhiy/py/cpython3.4/Lib', '/home/serhiy/py/cpython3.4/Lib/plat-linux', '/home/serhiy/py/cpython3.4/build/lib.linux-i686-3.4'])
  After:  (3071378764, ['', '/usr/local/lib/python34.zip', '/home/serhiy/py/cpython3.4/Lib', '/home/serhiy/py/cpython3.4/Lib/plat-linux', '/home/serhiy/py/cpython3.4/build/lib.linux-i686-3.4'], ['', '/usr/local/lib/python34.zip', '/home/serhiy/py/cpython3.4/Lib', '/home/serhiy/py/cpython3.4/Lib/plat-linux', '/home/serhiy/py/cpython3.4/build/lib.linux-i686-3.4', '/home/serhiy/.local/lib/python3.4/site-packages']) 
1 test altered the execution environment:
    test_site
msg256312 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-12-13 02:37
I can’t imagine why the buildbot is failing either.

The directory is created and added to sys.path when test_site is imported:

if site.ENABLE_USER_SITE and not os.path.isdir(site.USER_SITE):
    # need to add user site directory for tests
    os.makedirs(site.USER_SITE)
    site.addsitedir(site.USER_SITE)

Maybe it should be changed to skip test_s_option() if the directory is missing; I dunno. Another option might be to create it just for the test, and then remove it, but that risks interfering with concurrent tests.
msg289753 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2017-03-17 11:14
I just saw a similar failure on AppVeyor:

======================================================================
FAIL: test_s_option (test.test_site.HelperFunctionsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\cpython\lib\test\test_site.py", line 173, in test_s_option
    self.assertIn(usersite, sys.path)
AssertionError: 'C:\\Users\\appveyor\\AppData\\Roaming\\Python\\Python36\\site-packages' not found in ['', 'C:\\projects\\cpython\\PCbuild\\win32\\python36.zip', 'C:\\projects\\cpython\\DLLs', 'C:\\projects\\cpython\\lib', 'C:\\projects\\cpython\\PCbuild\\win32', 'C:\\projects\\cpython', 'C:\\projects\\cpython\\lib\\site-packages']

https://ci.appveyor.com/project/python/cpython/build/3.6.1rc1+.474#L3230
msg292732 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-02 09:45
It seems like the original bug report was fixed, so I close it.

@Berker Peksag: Please open a new bug report if you see again the PermissionError error on Windows. I didn't see it recently.


> Well, I'm not sure the symlink is actually the problem. Apparently the ".local" directory hadn't been created when the first test run.

I'm unable to reproduce this bug anymore. I named ~/.local to ~/.local.xxx: test_site pass (it starts by creating ~/.local/lib/python3.7/site-packages).

> I can't reproduce this locally. If remove ~/.local/lib/pythonX.Y and run test_site, it creates the directory and modifies sys.path (this is a bug).

I fixed this bug: commit b85c136903c6d2368162f7c4a58f258c9c69ead0, issue #30108.

> I just saw a similar failure on AppVeyor: FAIL: test_s_option (test.test_site.HelperFunctionsTests)

test_site failed twice. First run:

======================================================================
ERROR: test_underpth_file (test.test_site.StartupImportTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\cpython\lib\test\test_site.py", line 577, in test_underpth_file
    self._cleanup_underpth_exe(exe_file)
  File "C:\projects\cpython\lib\test\test_site.py", line 512, in _cleanup_underpth_exe
    test.support.unlink(exe_file)
  File "C:\projects\cpython\lib\test\support\__init__.py", line 398, in unlink
    _unlink(filename)
  File "C:\projects\cpython\lib\test\support\__init__.py", line 348, in _unlink
    _waitfor(os.unlink, filename)
  File "C:\projects\cpython\lib\test\support\__init__.py", line 316, in _waitfor
    func(pathname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\python.exe'
======================================================================
ERROR: test_underpth_nosite_file (test.test_site.StartupImportTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\cpython\lib\test\test_site.py", line 534, in test_underpth_nosite_file
    exe_file = self._create_underpth_exe(pth_lines)
  File "C:\projects\cpython\lib\test\test_site.py", line 495, in _create_underpth_exe
    shutil.copy(sys.executable, exe_file)
  File "C:\projects\cpython\lib\shutil.py", line 241, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "C:\projects\cpython\lib\shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\python.exe'


Second run:

======================================================================
FAIL: test_s_option (test.test_site.HelperFunctionsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\cpython\lib\test\test_site.py", line 173, in test_s_option
    self.assertIn(usersite, sys.path)
AssertionError: 'C:\\Users\\appveyor\\AppData\\Roaming\\Python\\Python36\\site-packages' not found in ['', 'C:\\projects\\cpython\\PCbuild\\win32\\python36.zip', 'C:\\projects\\cpython\\DLLs', 'C:\\projects\\cpython\\lib', 'C:\\projects\\cpython\\PCbuild\\win32', 'C:\\projects\\cpython', 'C:\\projects\\cpython\\lib\\site-packages']


The first failures are unrelated to this bug report. The first failure (PermissionError: [WinError 32] The process cannot access the file because it is being used by another process) seems like an antivirus holding the file open. Please open a different bug report if you see it again.
History
Date User Action Args
2017-05-02 09:45:25hayposetstatus: open -> closed

nosy: + haypo
messages: + msg292732

resolution: fixed
stage: resolved
2017-03-17 11:14:02berker.peksagsetnosy: + berker.peksag
messages: + msg289753
2015-12-13 02:37:20martin.pantersetnosy: + martin.panter
messages: + msg256312
2015-12-12 17:41:30serhiy.storchakasetstatus: closed -> open

versions: + Python 3.6
nosy: + serhiy.storchaka

messages: + msg256289
resolution: out of date -> (no value)
stage: resolved -> (no value)
2014-08-19 22:37:49ned.deilysetstatus: open -> closed
resolution: out of date
messages: + msg225546

stage: needs patch -> resolved
2014-06-22 22:23:17ned.deilysetnosy: + ned.deily
messages: + msg221320
2014-06-22 19:46:59yjchensetnosy: + yjchen
messages: + msg221301
2014-06-20 15:44:18zach.waresetassignee: tarek ->
versions: + Python 3.4, Python 3.5, - Python 3.2
2011-06-09 16:21:01merwoksetmessages: + msg138003
2010-11-22 04:45:27merwoksetnosy: + merwok
messages: + msg122093
2009-11-08 00:51:38pitrousettitle: test_site failure when Python gets installed along a symbolic link -> test_site failure when .local/lib/pythonX.Y/site-packages hasn't been created yet
2009-11-08 00:28:25pitrousetmessages: + msg95032
2009-11-08 00:27:12pitroucreate