classification
Title: tempfile.TemporaryFile fails when dir option set to directory residing on host OS mount
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Hans Lawrenz, Oleg Babintsev, abarry, bkabrda, martin.panter, pitrou, python-dev, serhiy.storchaka, vstinner
Priority: normal Keywords: 3.5regression, patch

Created on 2015-11-24 04:23 by Hans Lawrenz, last changed 2016-06-18 22:37 by Oleg Babintsev. This issue is now closed.

Files
File name Uploaded Description Edit
tempfile-strace.txt Hans Lawrenz, 2015-11-24 06:54
tempfile.py Hans Lawrenz, 2015-11-24 17:43
fstat-failure.patch martin.panter, 2015-11-24 22:10 review
Messages (27)
msg255246 - (view) Author: Hans Lawrenz (Hans Lawrenz) * Date: 2015-11-24 04:23
Inside a virtualbox vm, calling tempfile.TemporaryFile(dir=foo) where foo is a directory which resides on a volume mounted from the host OS, a FileNotFoundError exception is thrown.

In the following code sample, the second block will print "Path 2: ERROR" on Python 3.5 but not on 3.4 or 2.7 (assuming the vagrant environment provided below):

import tempfile


try:
    with tempfile.TemporaryFile() as tf:
        tf.write("testing testing testing\n".encode('utf-8'))
    print("Path 1: worked")
except FileNotFoundError as e:
    print("Path 1: ERROR")

try:
    with tempfile.TemporaryFile(dir="/vagrant") as tf:
        tf.write("testing testing testing\n".encode('utf-8'))
    print("Path 2: worked")
except FileNotFoundError as e:
    print("Path 2: ERROR")


A runnable test case can be found here:
https://github.com/hlawrenz/py35tempfiletest
msg255247 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-11-24 04:42
I am unable to produce this on native Linux so far with various kinds of mount points I normally have. What is your guest OS?

If you can provide any more details of the failure, such as a traceback, that would be helpful. What is the underlying OS call that is causing the exception?
msg255250 - (view) Author: Hans Lawrenz (Hans Lawrenz) * Date: 2015-11-24 06:54
Host OS: Mac OS 10.11.1
Guest OS: Ubuntu Trusty 64
Virtualbox: 4.3.30
Vagrant: 1.7.4


Example with trace thrown:

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ cat tt.py 
import tempfile


with tempfile.TemporaryFile(dir="/vagrant") as tf:
    tf.write("testing testing testing\n".encode('utf-8'))

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ python3.5 tt.py 
Traceback (most recent call last):
  File "tt.py", line 4, in <module>
    with tempfile.TemporaryFile(dir="/vagrant") as tf:
  File "/usr/lib/python3.5/tempfile.py", line 753, in TemporaryFile
    newline=newline, encoding=encoding)
FileNotFoundError: [Errno 2] No such file or directory


I think the underlying system call that's causing the error (I attached the full strace in case that's helpful):
open("/vagrant", O_RDWR|O_EXCL|O_DIRECTORY|O_NOFOLLOW|O_CLOEXEC|0x400000) = -1 EOPNOTSUPP (Operation not supported)

Finally, the mount point looks like this:
vagrant on /vagrant type vboxsf (uid=1000,gid=1000,rw)
msg255254 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-11-24 08:39
Thanks for the strace output. I think the actual error is the fstat() call a bit after that first open() call. FileNotFoundError is ENOENT:

open("/vagrant/tmpy5ioznh4", O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = 4
unlink("/vagrant/tmpy5ioznh4")          = 0
fstat(4, 0x7ffc0b326520)                = -1 ENOENT (No such file or directory)
close(4)                                = 0
write(2, "Traceback (most recent call last"..., 35Traceback (most recent call last):

My theory is that the fstat() call at <https://hg.python.org/cpython/annotate/3.5/Modules/_io/fileio.c#l441> is failing, probably because the file entry has been removed from its directory. This call was added by revision 3b5279b5bfd1 (Issue 21679). Before this change, fstat() was called twice, but in each case an error was tolerated.

Posix does not mention fstat() returning this ENOENT error, so maybe this could be a bug with the Virtual Box or Vagrant driver for the mounted filesystem. But it might be nice to make Python more permissive if fstat() fails, like it was in 3.4.

As a workaround, you may be able to use NamedTemporaryFile, if you are happy for the file to have a name and a directory entry.
msg255273 - (view) Author: Hans Lawrenz (Hans Lawrenz) * Date: 2015-11-24 15:38
Unfortunately changing the tempfile call isn't an easy option for me. The situation in which I'm encountering the error is running tox on our project which is mounted in the /vagrant directory in the VM (this is standard for vagrant). Tox makes its own temp directory--presumably for test isolation--in the .tox directory in the project root. The actual call to tempfile.TemporaryFile() which is triggering the error is in a third party library that is called during a test run. 

I was able to work around the issue by changing the tox workdir setting to outside the mount. However, I'd like to mention that developing in vagrant in this fashion isn't uncommon and once 3.5 gains adoption I would guess this issue may affect a good number of people.

I'm happy to take a stab at writing a patch but looking at the code it's somewhat out of my comfort zone and I worry I'd make a hash of it.
msg255275 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-11-24 15:51
What is exact version of you Python? I can't identify the line in the traceback. There are few lines "newline=newline, encoding=encoding)" in tempfile.py, but no one is closer to line 753.
msg255276 - (view) Author: Anilyka Barry (abarry) * (Python triager) Date: 2015-11-24 16:07
I'm with Serhiy, the line number is inane. Could you put the exact contents of your /usr/lib/python3.5/tempfile.py file somewhere for us to see? Output of sys.version would be nice, too.
msg255283 - (view) Author: Hans Lawrenz (Hans Lawrenz) * Date: 2015-11-24 17:43
Serhiy and Emanuel, I'll paste below the surrounding code and attach the exact tempfile.py. It is the version distributed with the 3.5.0 release. If you take a look at the github repo I linked in the first comment you can also try it out for yourself if you've got vagrant and virtualbox installed. I was able to recreate the error with the provisioning compiling python and with it installing from a ppa. You can see the details in the Vagrantfile. I also had a coworker test in a nearly identical environment and he had the same result.


        (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
        try:
            _os.unlink(name)
            return _io.open(fd, mode, buffering=buffering,
                            newline=newline, encoding=encoding)
        except:
            _os.close(fd)
            raise


One final note. I decided to try this out with a windows host and the only clarity this test adds is that the problem doesn't show up there but it fails for its own reasons on all the python versions I tried. See below for reference:

Last login: Tue Nov 24 17:16:12 2015 from 10.0.2.2
vagrant@vagrant-ubuntu-trusty-64:~$ cat /vagrant/foo.py
import tempfile

with tempfile.TemporaryFile(dir="/vagrant") as tf:
tf.write("testing testing testing\n".encode('utf-8'))
print("Path 2: worked")
vagrant@vagrant-ubuntu-trusty-64:~$ python /vagrant/foo.py
Traceback (most recent call last):
  File "/vagrant/foo.py", line 4, in <module>
    with tempfile.TemporaryFile(dir="/vagrant") as tf:
  File "/usr/lib/python2.7/tempfile.py", line 495, in TemporaryFile
    _os.unlink(name)
OSError: [Errno 26] Text file busy: '/vagrant/tmpvx7Mie'
vagrant@vagrant-ubuntu-trusty-64:~$ python2.7 /vagrant/foo.py
Traceback (most recent call last):
  File "/vagrant/foo.py", line 4, in <module>
    with tempfile.TemporaryFile(dir="/vagrant") as tf:
  File "/usr/lib/python2.7/tempfile.py", line 495, in TemporaryFile
    _os.unlink(name)
OSError: [Errno 26] Text file busy: '/vagrant/tmpNXQ6Cf'
vagrant@vagrant-ubuntu-trusty-64:~$ python3.4 /vagrant/foo.py
Traceback (most recent call last):
  File "/vagrant/foo.py", line 4, in <module>
    with tempfile.TemporaryFile(dir="/vagrant") as tf:
  File "/usr/lib/python3.4/tempfile.py", line 638, in TemporaryFile
    _os.unlink(name)
OSError: [Errno 26] Text file busy: '/vagrant/tmpfwhj7ip4'
vagrant@vagrant-ubuntu-trusty-64:~$ python3.5 /vagrant/foo.py
Traceback (most recent call last):
  File "/vagrant/foo.py", line 4, in <module>
    with tempfile.TemporaryFile(dir="/vagrant") as tf:
  File "/usr/lib/python3.5/tempfile.py", line 751, in TemporaryFile
    _os.unlink(name)
OSError: [Errno 26] Text file busy: '/vagrant/tmp02s19r_a'
vagrant@vagrant-ubuntu-trusty-64:~$ python2.7 --version
Python 2.7.6
vagrant@vagrant-ubuntu-trusty-64:~$ python3.4 --version
Python 3.4.3
vagrant@vagrant-ubuntu-trusty-64:~$ python3.5 --version
Python 3.5.0
msg255285 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-11-24 19:06
Is your file system NFS?
msg255286 - (view) Author: Anilyka Barry (abarry) * (Python triager) Date: 2015-11-24 19:38
Your file has a lot of shenanigans that are triggered if 'shutil' fails to be imported, adding an extra 139 lines at the top of the file, which is exactly how offset your traceback is compared to our lines. The rest of the file is virtually identical. This makes me wonder, however, why you have these differences (and that this might not be the only different file).

I would still like the output of sys.version, which, unlike 'python --version' contains the build date and time, among other things. Bonus points for the 'sys.implementation' output. It's unlikely to be of relevance, but I would rather explore every possibility.
msg255287 - (view) Author: Hans Lawrenz (Hans Lawrenz) * Date: 2015-11-24 19:40
The file system causing the problem is of type vboxsf which is the Virtualbox shared folder file system type.
msg255288 - (view) Author: Hans Lawrenz (Hans Lawrenz) * Date: 2015-11-24 19:58
Emanuel, sorry, I missed the request for sys.version earlier.

The tempfile.py I attached earlier is from the python 3.5 pulled from a ppa. I wouldn't be surprised if it has some patches applied by debian/ubuntu. To be clear though the problem also presents itself with a python 3.5 I compiled. The Vagrantfile in the github repository shows exactly how I compiled it.

Here it is from the version from the ppa:

Python 3.5.0 (default, Sep 17 2015, 00:00:00) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version
'3.5.0 (default, Sep 17 2015, 00:00:00) \n[GCC 4.8.4]'
>>> sys.implementation
namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-35', hexversion=50659568, name='cpython', version=sys.version_info(major=3, minor=5, micro=0, releaselevel='final', serial=0))


Here are the same for the version I compiled:
vagrant@vagrant-ubuntu-trusty-64:~$ python3.5
Python 3.5.0 (default, Nov 24 2015, 19:50:42) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version
'3.5.0 (default, Nov 24 2015, 19:50:42) \n[GCC 4.8.4]'
>>> sys.implementation
namespace(cache_tag='cpython-35', hexversion=50659568, name='cpython', version=sys.version_info(major=3, minor=5, micro=0, releaselevel='final', serial=0))
msg255292 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-11-24 22:10
This patch restores the previous behaviour of tolerating fstat() failures other than EBADF. Hans, if you are able to apply it to your compiled version of Python, it might prove that my fstat() theory is correct.
msg255352 - (view) Author: Hans Lawrenz (Hans Lawrenz) * Date: 2015-11-25 13:57
Martin, I applied your patch and it proved your hypothesis. See below for how I tested. I also updated the github repo for others to reproduce if they wish.


cd
wget https://www.python.org/ftp/python/3.5.0/Python-3.5.0.tar.xz

mkdir ~/dist
cd ~/dist
tar xJf ~/Python-3.5.0.tar.xz
cd Python-3.5.0
./configure --prefix=/home/vagrant/py35/dist && \
make && make install

mkdir ~/patch
cd ~/patch
tar xJf ~/Python-3.5.0.tar.xz
cd Python-3.5.0
patch -p1 < /vagrant/fstat-failure.patch
./configure --prefix=/home/vagrant/py35/patch && \
make && make install



vagrant@vagrant-ubuntu-trusty-64:~$ ./py35/dist/bin/python3.5 /vagrant/temptest.py 
Path 1: worked
Path 2: ERROR
vagrant@vagrant-ubuntu-trusty-64:~$ ./py35/patch/bin/python3.5 /vagrant/temptest.py 
Path 1: worked
Path 2: worked
msg255553 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-11-28 22:23
Antoine or Bohuslav, since you were involved in Issue 21679, would you be able to comment on fstat-failure.patch, which reverts back to ignoring most fstat() errors? The problem here is that fstat() seems to fail if the directory entry has been unlinked, and the file is on a Virtual Box shared folder filesystem.
msg255647 - (view) Author: Bohuslav "Slavek" Kabrda (bkabrda) * Date: 2015-12-01 11:00
The patch looks good to me.
msg256001 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-12-06 03:51
New changeset 20ea12222b0e by Martin Panter in branch '3.5':
Issue #25717: Tolerate fstat() failures in the FileIO constructor
https://hg.python.org/cpython/rev/20ea12222b0e

New changeset 8c978cbe057c by Martin Panter in branch 'default':
Issue #25717: Merge fstat() fix from 3.5
https://hg.python.org/cpython/rev/8c978cbe057c
msg256002 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-12-06 05:01
Thanks Bohuslav, and also to Hans for helping track this down.
msg256006 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-12-06 08:33
Martin, please add a comment to explain the rationale on ignoring errors
other than EBADF. At least, mention this issue number.
msg256008 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-12-06 11:15
Okay will do
msg256009 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-12-06 11:20
New changeset e9bf5803b716 by Martin Panter in branch '3.5':
Issue #25717: Add comment explaining why errors are ignored
https://hg.python.org/cpython/rev/e9bf5803b716

New changeset 8bf69413ec01 by Martin Panter in branch 'default':
Issue #25717: Merge comment from 3.5
https://hg.python.org/cpython/rev/8bf69413ec01
msg256015 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-12-06 15:30
Thank you.
msg268783 - (view) Author: Oleg Babintsev (Oleg Babintsev) Date: 2016-06-18 09:40
Problem is still actual for Python 2.7
Can you provide the fix for 2.7 too?
msg268790 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-06-18 09:59
Are you sure Oleg? As far as I understand, Python 2 by default wraps C stdio file objects, and also has Python 3’s file objects in the “io” module. But I expect TemporaryFile() would use the default stdio files, and the cause of this bug, Issue 21679, should only have affected the “io” module in 3.5+.
msg268792 - (view) Author: Oleg Babintsev (Oleg Babintsev) Date: 2016-06-18 10:35
My environment: 
Windows as host OS, VirtualBox VM, Linux as guest OS

When temporary directory is a directory which mounted from the host OS, a "OSError: [Errno 26] Text file busy" exception is thrown.

Test file:

import tempfile

with tempfile.TemporaryFile(dir="/tmp") as tf:
    tf.write("testing testing testing\n".encode('utf-8'))
print("Temp file: worked")

"/tmp" directory - is a shared folder (vboxsf)

Result:

0e87f2561643">root@0e87f2561643:/# python /etc/odoo/tempfile-test.py
Traceback (most recent call last):
  File "/etc/odoo/tempfile-test.py", line 3, in <module>
    with tempfile.TemporaryFile(dir="/tmp") as tf:
  File "/usr/lib/python2.7/tempfile.py", line 513, in TemporaryFile
    _os.unlink(name)
OSError: [Errno 26] Text file busy: '/tmp/tmpsl0JQI'

0e87f2561643">root@0e87f2561643:/# python --version
Python 2.7.11+

0e87f2561643">root@0e87f2561643:/# python
Python 2.7.11+ (default, Jun  2 2016, 19:34:15)
[GCC 5.3.1 20160528] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version
'2.7.11+ (default, Jun  2 2016, 19:34:15) \n[GCC 5.3.1 20160528]'
>>>
msg268796 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-06-18 11:36
This original bug was about fstat() failing for an anonymous file (after the directory entry was removed). But in your situation the file system refuses to unlink the directory entry.

If you think there is something that can be fixed in Python, I suggest open a new bug. But it seems this is working as documented: “Under Unix, the directory entry for the file is removed immediately after the file is created.” In your case, this cannot happen, so you get the error from the OS.
msg268834 - (view) Author: Oleg Babintsev (Oleg Babintsev) Date: 2016-06-18 22:37
Thanks for explanation.
History
Date User Action Args
2016-06-18 22:37:07Oleg Babintsevsetmessages: + msg268834
2016-06-18 11:36:33martin.pantersetmessages: + msg268796
versions: + Python 3.5, Python 3.6, - Python 2.7
2016-06-18 10:35:39Oleg Babintsevsetmessages: + msg268792
2016-06-18 09:59:53martin.pantersetmessages: + msg268790
2016-06-18 09:40:25Oleg Babintsevsetnosy: + Oleg Babintsev

messages: + msg268783
versions: + Python 2.7, - Python 3.5, Python 3.6
2016-02-11 01:43:11eryksunlinkissue25639 superseder
2015-12-06 15:30:20vstinnersetmessages: + msg256015
2015-12-06 11:20:34python-devsetmessages: + msg256009
2015-12-06 11:15:29martin.pantersetmessages: + msg256008
2015-12-06 08:33:34vstinnersetmessages: + msg256006
2015-12-06 05:01:41martin.pantersetstatus: open -> closed
resolution: fixed
messages: + msg256002

stage: patch review -> resolved
2015-12-06 03:51:58python-devsetnosy: + python-dev
messages: + msg256001
2015-12-01 11:00:16bkabrdasetmessages: + msg255647
2015-11-28 22:23:47martin.pantersetversions: + Python 3.6
nosy: + pitrou, bkabrda

messages: + msg255553

stage: patch review
2015-11-25 13:57:02Hans Lawrenzsetmessages: + msg255352
2015-11-24 22:16:03serhiy.storchakasetnosy: + vstinner
2015-11-24 22:10:46martin.pantersetfiles: + fstat-failure.patch
keywords: + patch
messages: + msg255292
2015-11-24 19:58:27Hans Lawrenzsetmessages: + msg255288
2015-11-24 19:40:55Hans Lawrenzsetmessages: + msg255287
2015-11-24 19:38:21abarrysetmessages: + msg255286
2015-11-24 19:06:27serhiy.storchakasetmessages: + msg255285
2015-11-24 17:43:56Hans Lawrenzsetfiles: + tempfile.py

messages: + msg255283
2015-11-24 16:07:13abarrysetnosy: + abarry
messages: + msg255276
2015-11-24 15:51:19serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg255275
2015-11-24 15:38:38Hans Lawrenzsetmessages: + msg255273
2015-11-24 08:39:42martin.pantersetmessages: + msg255254
2015-11-24 06:54:31Hans Lawrenzsetfiles: + tempfile-strace.txt

messages: + msg255250
2015-11-24 04:42:49martin.pantersetkeywords: + 3.5regression
nosy: + martin.panter
messages: + msg255247

2015-11-24 04:23:44Hans Lawrenzcreate