classification
Title: test_compileall fails to build in the installed location
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: duplicate
Dependencies: Superseder: test_compileall fails when run by unprivileged user on installed Python
View: 25099
Assigned To: brett.cannon Nosy List: brett.cannon, doko, ezio.melotti, michael.foord, r.david.murray, terry.reedy
Priority: normal Keywords: patch

Created on 2014-04-16 18:59 by doko, last changed 2015-10-09 22:11 by brett.cannon. This issue is now closed.

Files
File name Uploaded Description Edit
issue21264.diff doko, 2014-05-06 12:57
Messages (8)
msg216553 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2014-04-16 18:59
the installation directory is non-writable, and the byte code files don't exist.

[1/1] test_compileall
test test_compileall failed -- Traceback (most recent call last):
  File "/usr/lib/python3.4/test/test_compileall.py", line 194, in test_no_args_respects_force_flag
    self.assertRunOK('-f', PYTHONPATH=self.directory)
  File "/usr/lib/python3.4/test/test_compileall.py", line 144, in assertRunOK
    *self._get_run_args(args), **env_vars)
  File "/usr/lib/python3.4/test/script_helper.py", line 69, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/usr/lib/python3.4/test/script_helper.py", line 55, in _assert_python
    "stderr follows:\n%s" % (rc, err.decode('ascii', 'ignore')))
AssertionError: Process return code is 1, stderr follows:


1 test failed:
    test_compileall
Re-running failed tests in verbose mode
Re-running test 'test_compileall' in verbose mode

======================================================================
FAIL: test_no_args_respects_force_flag (test.test_compileall.CommandLineTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.4/test/test_compileall.py", line 194, in test_no_args_respects_force_flag
    self.assertRunOK('-f', PYTHONPATH=self.directory)
  File "/usr/lib/python3.4/test/test_compileall.py", line 144, in assertRunOK
    *self._get_run_args(args), **env_vars)
  File "/usr/lib/python3.4/test/script_helper.py", line 69, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/usr/lib/python3.4/test/script_helper.py", line 55, in _assert_python
    "stderr follows:\n%s" % (rc, err.decode('ascii', 'ignore')))
AssertionError: Process return code is 1, stderr follows:
msg217979 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2014-05-06 12:57
what happens here:

PYTHONPATH=$(pwd) python3.4 -X faulthandler -S -m compileall
Skipping current directory
Listing '/home/packages/python/3.4/x'...
Compiling '/home/packages/python/3.4/x/foo.py'...
Listing '/usr/lib/python3.4/'...
Listing '/usr/lib/python3.4/plat-x86_64-linux-gnu'...
Listing '/usr/lib/python3.4/lib-dynload'...

so even when overwriting PYTHONPATH, the standard locations are searched and tried to compile, which fails when these are read only.

Is this behaviour expected?

The tests itself can be fixed by passing the the test directory as an argument, as done in other tests too.
msg218085 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2014-05-07 22:01
adding unittest developers
msg251031 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-09-18 21:46
Verified failure on installed 3.4.3, Win 7, with 3 rather than 1 error (all similar).
test_no_args_compiles_path (test.test_compileall.CommandLineTests) ... FAIL
test_no_args_respects_force_flag (test.test_compileall.CommandLineTests) ... FAIL
test_no_args_respects_quiet_flag (test.test_compileall.CommandLineTests) ... FAIL
test_pep3147_paths_doubleoptimize (test.test_compileall.CommandLineTests) ... ok

First traceback ends with
AssertionError: Process return code is 1, command line was: ['C:\\Programs\\Python34\\python.exe', '-X', 'faulthandler', '-S', '-m', 'compil
eall'], stderr follows:

followed by another ending with
  File "C:\Programs\Python34\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\U00011111' in position 38: character maps to <undefined>

Why is an astral char in the error message?

Will try patch.
msg251039 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-09-18 23:04
David, I am nosying you because this issue proposes to modify two test functions you added to test_compileall. See my prior message.
--
For me, the test failed at every first occurrence of 'PYTHONPATH=' in the 3 functions.  With one occurrence removed, the next in the same function failed. With all 6 removed, I get 1 error:

FAIL: test_no_args_compiles_path (test.test_compileall.CommandLineTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Programs\Python34\lib\test\test_compileall.py", line 179, in test_no_args_compiles_path
    self.assertNotCompiled(self.initfn)
  File "C:\Programs\Python34\lib\test\test_compileall.py", line 159, in assertNotCompiled
    self.assertFalse(os.path.exists(path))
AssertionError: True is not false

The tests all pass in current uninstalled 3.4.3+ repository. If I remove all 6, I get the error above. Changing how the directory is passed apparently changes what get compiled. From the title of the test ad the comment "# Note that -l is implied for the no args case.", this is expected and is the point of the test.  So deleting 'PYTHONPATH=' must be wrong.  I believe we should detect that modifying the environment will not work before calling assertRunOK and skip the latter if it cannot work.

The other two functions were added in #19532 by RDM. I think we need to know why they were written as they are before changing them. Perhaps they pass with 'PYTHONPATH=' removed because they are incomplete.
msg251099 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-09-19 18:45
Yes the point is to test compilation of the path.  If I run python3 -m compilelall on my installed python, it works fine (zero return code).  I can't really investigate this further right now, but it isn't obvious to me what the answer is from the information in the issue so far, except that it doesn't *look* like it should be necessary to skip the tests unless the installation directories are read only *and* the .pyc files in them are non-existent or out of date.
msg251610 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-09-25 20:53
Issue #25099 seems to also be related to a similar issue. A proposed fix can be found over there.
msg252670 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-10-09 22:11
I consider this fixed by way of issue #25099. If that's wrong then feel free to open the issue again.
History
Date User Action Args
2015-10-09 22:11:37brett.cannonsetstatus: open -> closed
versions: + Python 3.5, Python 3.6
superseder: test_compileall fails when run by unprivileged user on installed Python
messages: + msg252670

resolution: duplicate
stage: patch review -> resolved
2015-09-25 20:54:01brett.cannonsetassignee: brett.cannon
2015-09-25 20:53:54brett.cannonsetnosy: + brett.cannon
messages: + msg251610
2015-09-19 18:45:03r.david.murraysetmessages: + msg251099
2015-09-18 23:04:41terry.reedysetnosy: + r.david.murray
messages: + msg251039
2015-09-18 21:46:39terry.reedysettype: behavior

messages: + msg251031
nosy: + terry.reedy
2014-05-07 22:01:04dokosetnosy: + ezio.melotti, michael.foord
messages: + msg218085
2014-05-06 12:57:28dokosetfiles: + issue21264.diff
messages: + msg217979

components: + Tests
keywords: + patch
stage: patch review
2014-04-16 19:00:12dokolinkissue17750 dependencies
2014-04-16 18:59:40dokocreate