classification
Title: test_relative_path of test_py_compile fails on macOS 10.15 Catalina
Type: behavior Stage: patch review
Components: macOS, Tests Versions: Python 3.9, Python 3.8, Python 3.7, Python 3.6, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Bo98, amgedr, barry, maggyero, miss-islington, ned.deily, ronaldoussoren, terry.reedy
Priority: normal Keywords: patch

Created on 2019-09-27 17:24 by barry, last changed 2019-12-19 01:50 by ned.deily.

Pull Requests
URL Status Linked Edit
PR 17636 merged ned.deily, 2019-12-17 08:47
PR 17637 merged miss-islington, 2019-12-17 09:06
PR 17638 merged miss-islington, 2019-12-17 09:06
PR 17639 merged miss-islington, 2019-12-17 09:06
Messages (13)
msg353381 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2019-09-27 17:24
test_py_compile fails on macOS Catalina beta (19A573a)

======================================================================
ERROR: test_relative_path (test.test_py_compile.PyCompileTestsWithSourceEpoch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bwarsaw/projects/python/3.8/Lib/test/test_py_compile.py", line 30, in wrapper
    return fxn(*args, **kwargs)
  File "/Users/bwarsaw/projects/python/3.8/Lib/test/test_py_compile.py", line 111, in test_relative_path
    py_compile.compile(os.path.relpath(self.source_path),
  File "/Users/bwarsaw/projects/python/3.8/Lib/py_compile.py", line 157, in compile
    os.makedirs(dirname)
  File "/Users/bwarsaw/projects/python/3.8/Lib/os.py", line 221, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '../../../../../../../var/folders/w6/w_2zrjgj7bgdgkhdjm4_r9s4000slb/T/tmpq3p6aoly'

======================================================================
ERROR: test_relative_path (test.test_py_compile.PyCompileTestsWithoutSourceEpoch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bwarsaw/projects/python/3.8/Lib/test/test_py_compile.py", line 20, in wrapper
    return fxn(*args, **kwargs)
  File "/Users/bwarsaw/projects/python/3.8/Lib/test/test_py_compile.py", line 111, in test_relative_path
    py_compile.compile(os.path.relpath(self.source_path),
  File "/Users/bwarsaw/projects/python/3.8/Lib/py_compile.py", line 157, in compile
    os.makedirs(dirname)
  File "/Users/bwarsaw/projects/python/3.8/Lib/os.py", line 221, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '../../../../../../../var/folders/w6/w_2zrjgj7bgdgkhdjm4_r9s4000slb/T/tmpquor0q8r'

----------------------------------------------------------------------
msg353408 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-09-27 20:33
This is not a duplicate because Python *did* compile enough to fail a test.
msg354022 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-10-05 21:25
Also the similar single test in 2.7.x fails on 10.15 (pre-release):

======================================================================
ERROR: test_relative_path (test.test_py_compile.PyCompileTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/test/test_py_compile.py", line 45, in test_relative_path
    os.path.relpath(self.pyc_path))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/py_compile.py", line 123, in compile
    with open(cfile, 'wb') as fc:
IOError: [Errno 2] No such file or directory: '../../../../../../../var/folders/n6/q3475zjd38bc3q197d77wjc80000gn/T/tmpLPxkJN/_test.pyc'
msg357483 - (view) Author: Bo Anderson (Bo98) Date: 2019-11-26 05:17
For what it's worth, this is having an impact on some real code: https://github.com/Homebrew/homebrew-core/pull/45110

Perhaps a simpler way to reproduce is:

% cd /tmp
% python3 -c 'import os; open(os.path.relpath("/tmp/test.txt"), "w")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: '../../tmp/test.txt'
msg358535 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-12-17 09:05
New changeset bf3aa1060a29a05813abbe877193af16e3e7131e by Ned Deily in branch '3.8':
bpo-38295: prevent test_relative_path of test_py_compile failure on macOS Catalina (GH-17636)
https://github.com/python/cpython/commit/bf3aa1060a29a05813abbe877193af16e3e7131e
msg358536 - (view) Author: miss-islington (miss-islington) Date: 2019-12-17 09:16
New changeset 5f2c1345a79f205c680ed6e0a6ed44199546d79e by Miss Islington (bot) in branch '2.7':
bpo-38295: prevent test_relative_path of test_py_compile failure on macOS Catalina (GH-17636)
https://github.com/python/cpython/commit/5f2c1345a79f205c680ed6e0a6ed44199546d79e
msg358537 - (view) Author: miss-islington (miss-islington) Date: 2019-12-17 09:24
New changeset 3dbfe0ac93ce031a23342a62edd39506bcdaea0e by Miss Islington (bot) in branch '3.7':
bpo-38295: prevent test_relative_path of test_py_compile failure on macOS Catalina (GH-17636)
https://github.com/python/cpython/commit/3dbfe0ac93ce031a23342a62edd39506bcdaea0e
msg358538 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-12-17 09:26
New changeset 5a9b69732ec19b04de459af890bcc26b83d979c5 by Ned Deily (Miss Islington (bot)) in branch '3.6':
bpo-38295: prevent test_relative_path of test_py_compile failure on macOS Catalina (GH-17636) (GH-17638)
https://github.com/python/cpython/commit/5a9b69732ec19b04de459af890bcc26b83d979c5
msg358539 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-12-17 09:39
For master and 3.9.0a2, equivalent workaround was included in a76ba362c4d86adf5e7f8254398135d12d7afd25 (bpo-39041 / GH-17594).

While these changes prevent the specific test failure, they point to more general issues with use of relative paths on macOS particularly with temp files and with 10.15 Catalina. More a bit later ...
msg358541 - (view) Author: Bo Anderson (Bo98) Date: 2019-12-17 10:59
Indeed. The issue can be trivially reproduced with:

```
#include <fcntl.h>
#include <errno.h>
#include <stdio.h>
#include <unistd.h>

int main()
{
  char buf[255];
  printf("Current dir: %s\n", getcwd(buf, 255));

  int fd = open("../../tmp/test.txt", O_WRONLY | O_CREAT);
  if (fd < 0)
  {
    printf("errno %d\n", errno);
    return 1;
  }
  close(fd);
  printf("Success\n");
  return 0;
}
```

and running it in /private/tmp.

I filed FB7467762 at the end of November. Downstream projects meanwhile are working around the issue by resolving the file path before passing it into `open`.
msg358546 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2019-12-17 14:41
You don't even need a C program to reproduce:

$ cd /private/tmp
$ touch ../../foo.txt
touch: ../../tmp/foo.txt: No such file or directory

And the "cd" is optional, I get the same error from my home directory (/Users/ronald). 

All of this on macOS 10.15.2
msg358550 - (view) Author: Bo Anderson (Bo98) Date: 2019-12-17 15:36
> You don't even need a C program to reproduce

Indeed, touch is built upon the POSIX file API (unless Apple modified it). The idea for the Apple bug report was to show it happening at a low level and not specific to a given tool.

> And the "cd" is optional, I get the same error from my home directory

Yes, /private/tmp is just an example but I'd be cautious saying the same happens everywhere. You won't be able to reproduce the issue if your current directory is /usr.

/private/tmp, your home directory, etc. are all "firmlinked" to /System/Volumes/Data in Catalina. /System/Volumes/Data/private/tmp exists but /System/Volumes/Data/tmp doesn't exist. This shouldn't really be an issue as the idea of firmlinks is to make the /System/Volumes/Data invisible and thus you should be able to relatively go back up to the /System/Volumes/Data and be transported back to the root directory, where you can find the /tmp symlink (and indeed that works fine with `ls`). Evidently that doesn't seem to work properly for file operations.
msg358651 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-12-19 01:50
New changeset 13ee023c03caf85101778b9323cdffbad695a4e0 by Ned Deily (Miss Islington (bot)) in branch '3.7':
bpo-38295: prevent test_relative_path of test_py_compile failure on macOS Catalina (GH-17636)
https://github.com/python/cpython/commit/13ee023c03caf85101778b9323cdffbad695a4e0
History
Date User Action Args
2019-12-19 01:50:10ned.deilysetmessages: + msg358651
2019-12-17 15:36:47Bo98setmessages: + msg358550
2019-12-17 14:41:27ronaldoussorensetmessages: + msg358546
2019-12-17 10:59:08Bo98setmessages: + msg358541
2019-12-17 09:39:33ned.deilysetmessages: + msg358539
2019-12-17 09:26:15ned.deilysetmessages: + msg358538
2019-12-17 09:24:02miss-islingtonsetmessages: + msg358537
2019-12-17 09:16:49miss-islingtonsetnosy: + miss-islington
messages: + msg358536
2019-12-17 09:06:24miss-islingtonsetpull_requests: + pull_request17107
2019-12-17 09:06:16miss-islingtonsetpull_requests: + pull_request17106
2019-12-17 09:06:09miss-islingtonsetpull_requests: + pull_request17105
2019-12-17 09:05:49ned.deilysetmessages: + msg358535
2019-12-17 08:47:54ned.deilysetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request17104
2019-12-14 19:16:23maggyerosetnosy: + maggyero
2019-12-11 06:36:54ned.deilylinkissue39024 superseder
2019-12-03 17:29:48amgedrsetnosy: + amgedr
2019-11-26 05:17:14Bo98setnosy: + Bo98
messages: + msg357483
2019-10-05 21:25:52ned.deilysettitle: macOS Catalina test failures -> test_relative_path of test_py_compile fails on macOS 10.15 Catalina
messages: + msg354022
versions: + Python 2.7
2019-09-27 20:33:25terry.reedysetstatus: closed -> open

superseder: ./configure on recent macOS versions can fail with multiple "present but cannot be compiled" messages ->
nosy: + terry.reedy, ronaldoussoren
components: + macOS

messages: + msg353408
type: compile error -> behavior
resolution: duplicate ->
stage: resolved -> needs patch
2019-09-27 20:28:47terry.reedysetstatus: open -> closed
type: compile error
superseder: ./configure on recent macOS versions can fail with multiple "present but cannot be compiled" messages
resolution: duplicate
stage: resolved
2019-09-27 17:24:03barrycreate