test_py_compile fails on macOS Catalina beta (19A573a)

======================================================================
ERROR: test_relative_path (test.test_py_compile.PyCompileTestsWithSourceEpoch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bwarsaw/projects/python/3.8/Lib/test/test_py_compile.py", line 30, in wrapper
    return fxn(*args, **kwargs)
  File "/Users/bwarsaw/projects/python/3.8/Lib/test/test_py_compile.py", line 111, in test_relative_path
    py_compile.compile(os.path.relpath(self.source_path),
  File "/Users/bwarsaw/projects/python/3.8/Lib/py_compile.py", line 157, in compile
    os.makedirs(dirname)
  File "/Users/bwarsaw/projects/python/3.8/Lib/os.py", line 221, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '../../../../../../../var/folders/w6/w_2zrjgj7bgdgkhdjm4_r9s4000slb/T/tmpq3p6aoly'

======================================================================
ERROR: test_relative_path (test.test_py_compile.PyCompileTestsWithoutSourceEpoch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bwarsaw/projects/python/3.8/Lib/test/test_py_compile.py", line 20, in wrapper
    return fxn(*args, **kwargs)
  File "/Users/bwarsaw/projects/python/3.8/Lib/test/test_py_compile.py", line 111, in test_relative_path
    py_compile.compile(os.path.relpath(self.source_path),
  File "/Users/bwarsaw/projects/python/3.8/Lib/py_compile.py", line 157, in compile
    os.makedirs(dirname)
  File "/Users/bwarsaw/projects/python/3.8/Lib/os.py", line 221, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '../../../../../../../var/folders/w6/w_2zrjgj7bgdgkhdjm4_r9s4000slb/T/tmpquor0q8r'

----------------------------------------------------------------------

This is not a duplicate because Python *did* compile enough to fail a test.

Also the similar single test in 2.7.x fails on 10.15 (pre-release):

======================================================================
ERROR: test_relative_path (test.test_py_compile.PyCompileTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/test/test_py_compile.py", line 45, in test_relative_path
    os.path.relpath(self.pyc_path))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/py_compile.py", line 123, in compile
    with open(cfile, 'wb') as fc:
IOError: [Errno 2] No such file or directory: '../../../../../../../var/folders/n6/q3475zjd38bc3q197d77wjc80000gn/T/tmpLPxkJN/_test.pyc'

For what it's worth, this is having an impact on some real code: https://github.com/Homebrew/homebrew-core/pull/45110

Perhaps a simpler way to reproduce is:

% cd /tmp
% python3 -c 'import os; open(os.path.relpath("/tmp/test.txt"), "w")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: '../../tmp/test.txt'

New changeset bf3aa1060a29a05813abbe877193af16e3e7131e by Ned Deily in branch '3.8':
bpo-38295: prevent test_relative_path of test_py_compile failure on macOS Catalina (GH-17636)
https://github.com/python/cpython/commit/bf3aa1060a29a05813abbe877193af16e3e7131e

New changeset 5f2c1345a79f205c680ed6e0a6ed44199546d79e by Miss Islington (bot) in branch '2.7':
bpo-38295: prevent test_relative_path of test_py_compile failure on macOS Catalina (GH-17636)
https://github.com/python/cpython/commit/5f2c1345a79f205c680ed6e0a6ed44199546d79e

New changeset 3dbfe0ac93ce031a23342a62edd39506bcdaea0e by Miss Islington (bot) in branch '3.7':
bpo-38295: prevent test_relative_path of test_py_compile failure on macOS Catalina (GH-17636)
https://github.com/python/cpython/commit/3dbfe0ac93ce031a23342a62edd39506bcdaea0e

New changeset 5a9b69732ec19b04de459af890bcc26b83d979c5 by Ned Deily (Miss Islington (bot)) in branch '3.6':
bpo-38295: prevent test_relative_path of test_py_compile failure on macOS Catalina (GH-17636) (GH-17638)
https://github.com/python/cpython/commit/5a9b69732ec19b04de459af890bcc26b83d979c5

For master and 3.9.0a2, equivalent workaround was included in a76ba362c4d86adf5e7f8254398135d12d7afd25 (bpo-39041 / GH-17594).

While these changes prevent the specific test failure, they point to more general issues with use of relative paths on macOS particularly with temp files and with 10.15 Catalina. More a bit later ...

Indeed. The issue can be trivially reproduced with:

```
#include <fcntl.h>
#include <errno.h>
#include <stdio.h>
#include <unistd.h>

int main()
{
  char buf[255];
  printf("Current dir: %s\n", getcwd(buf, 255));

  int fd = open("../../tmp/test.txt", O_WRONLY | O_CREAT);
  if (fd < 0)
  {
    printf("errno %d\n", errno);
    return 1;
  }
  close(fd);
  printf("Success\n");
  return 0;
}
```

and running it in /private/tmp.

I filed FB7467762 at the end of November. Downstream projects meanwhile are working around the issue by resolving the file path before passing it into `open`.

You don't even need a C program to reproduce:

$ cd /private/tmp
$ touch ../../foo.txt
touch: ../../tmp/foo.txt: No such file or directory

And the "cd" is optional, I get the same error from my home directory (/Users/ronald). 

All of this on macOS 10.15.2

> You don't even need a C program to reproduce

Indeed, touch is built upon the POSIX file API (unless Apple modified it). The idea for the Apple bug report was to show it happening at a low level and not specific to a given tool.

> And the "cd" is optional, I get the same error from my home directory

Yes, /private/tmp is just an example but I'd be cautious saying the same happens everywhere. You won't be able to reproduce the issue if your current directory is /usr.

/private/tmp, your home directory, etc. are all "firmlinked" to /System/Volumes/Data in Catalina. /System/Volumes/Data/private/tmp exists but /System/Volumes/Data/tmp doesn't exist. This shouldn't really be an issue as the idea of firmlinks is to make the /System/Volumes/Data invisible and thus you should be able to relatively go back up to the /System/Volumes/Data and be transported back to the root directory, where you can find the /tmp symlink (and indeed that works fine with `ls`). Evidently that doesn't seem to work properly for file operations.

New changeset 13ee023c03caf85101778b9323cdffbad695a4e0 by Ned Deily (Miss Islington (bot)) in branch '3.7':
bpo-38295: prevent test_relative_path of test_py_compile failure on macOS Catalina (GH-17636)
https://github.com/python/cpython/commit/13ee023c03caf85101778b9323cdffbad695a4e0

Is there anything more to do here?

> Is there anything more to do here?

A belated response: probably not, so I'm OK with closing this.

A quick check shows that macOS behavior has changed a bit on more recent releases. On current macOS Big Sur (11.6.3) and Monterey (12.2) releases, Bo's test:

$ python3 -c 'import os; open(os.path.relpath("/tmp/test.txt"), "w")'

now works and Ronald's touch example now fails with a somewhat more meaningful message:

$ touch ../../foo.txt
touch: ../../foo.txt: Read-only file system

In general, though, even on pre-APFS older macOS systems, it is still best to avoid or at least be very careful about using /tmp with any sort of relative path testing because of the linking of /tmp to /private/tmp. :(