Issue25717
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2015-11-24 04:23 by Hans Lawrenz, last changed 2022-04-11 14:58 by admin. This issue is now closed.
Files | ||||
---|---|---|---|---|
File name | Uploaded | Description | Edit | |
tempfile-strace.txt | Hans Lawrenz, 2015-11-24 06:54 | |||
tempfile.py | Hans Lawrenz, 2015-11-24 17:43 | |||
fstat-failure.patch | martin.panter, 2015-11-24 22:10 | review |
Messages (27) | |||
---|---|---|---|
msg255246 - (view) | Author: Hans Lawrenz (Hans Lawrenz) * | Date: 2015-11-24 04:23 | |
Inside a virtualbox vm, calling tempfile.TemporaryFile(dir=foo) where foo is a directory which resides on a volume mounted from the host OS, a FileNotFoundError exception is thrown. In the following code sample, the second block will print "Path 2: ERROR" on Python 3.5 but not on 3.4 or 2.7 (assuming the vagrant environment provided below): import tempfile try: with tempfile.TemporaryFile() as tf: tf.write("testing testing testing\n".encode('utf-8')) print("Path 1: worked") except FileNotFoundError as e: print("Path 1: ERROR") try: with tempfile.TemporaryFile(dir="/vagrant") as tf: tf.write("testing testing testing\n".encode('utf-8')) print("Path 2: worked") except FileNotFoundError as e: print("Path 2: ERROR") A runnable test case can be found here: https://github.com/hlawrenz/py35tempfiletest |
|||
msg255247 - (view) | Author: Martin Panter (martin.panter) * ![]() |
Date: 2015-11-24 04:42 | |
I am unable to produce this on native Linux so far with various kinds of mount points I normally have. What is your guest OS? If you can provide any more details of the failure, such as a traceback, that would be helpful. What is the underlying OS call that is causing the exception? |
|||
msg255250 - (view) | Author: Hans Lawrenz (Hans Lawrenz) * | Date: 2015-11-24 06:54 | |
Host OS: Mac OS 10.11.1 Guest OS: Ubuntu Trusty 64 Virtualbox: 4.3.30 Vagrant: 1.7.4 Example with trace thrown: vagrant@vagrant-ubuntu-trusty-64:/vagrant$ cat tt.py import tempfile with tempfile.TemporaryFile(dir="/vagrant") as tf: tf.write("testing testing testing\n".encode('utf-8')) vagrant@vagrant-ubuntu-trusty-64:/vagrant$ python3.5 tt.py Traceback (most recent call last): File "tt.py", line 4, in <module> with tempfile.TemporaryFile(dir="/vagrant") as tf: File "/usr/lib/python3.5/tempfile.py", line 753, in TemporaryFile newline=newline, encoding=encoding) FileNotFoundError: [Errno 2] No such file or directory I think the underlying system call that's causing the error (I attached the full strace in case that's helpful): open("/vagrant", O_RDWR|O_EXCL|O_DIRECTORY|O_NOFOLLOW|O_CLOEXEC|0x400000) = -1 EOPNOTSUPP (Operation not supported) Finally, the mount point looks like this: vagrant on /vagrant type vboxsf (uid=1000,gid=1000,rw) |
|||
msg255254 - (view) | Author: Martin Panter (martin.panter) * ![]() |
Date: 2015-11-24 08:39 | |
Thanks for the strace output. I think the actual error is the fstat() call a bit after that first open() call. FileNotFoundError is ENOENT: open("/vagrant/tmpy5ioznh4", O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = 4 unlink("/vagrant/tmpy5ioznh4") = 0 fstat(4, 0x7ffc0b326520) = -1 ENOENT (No such file or directory) close(4) = 0 write(2, "Traceback (most recent call last"..., 35Traceback (most recent call last): My theory is that the fstat() call at <https://hg.python.org/cpython/annotate/3.5/Modules/_io/fileio.c#l441> is failing, probably because the file entry has been removed from its directory. This call was added by revision 3b5279b5bfd1 (Issue 21679). Before this change, fstat() was called twice, but in each case an error was tolerated. Posix does not mention fstat() returning this ENOENT error, so maybe this could be a bug with the Virtual Box or Vagrant driver for the mounted filesystem. But it might be nice to make Python more permissive if fstat() fails, like it was in 3.4. As a workaround, you may be able to use NamedTemporaryFile, if you are happy for the file to have a name and a directory entry. |
|||
msg255273 - (view) | Author: Hans Lawrenz (Hans Lawrenz) * | Date: 2015-11-24 15:38 | |
Unfortunately changing the tempfile call isn't an easy option for me. The situation in which I'm encountering the error is running tox on our project which is mounted in the /vagrant directory in the VM (this is standard for vagrant). Tox makes its own temp directory--presumably for test isolation--in the .tox directory in the project root. The actual call to tempfile.TemporaryFile() which is triggering the error is in a third party library that is called during a test run. I was able to work around the issue by changing the tox workdir setting to outside the mount. However, I'd like to mention that developing in vagrant in this fashion isn't uncommon and once 3.5 gains adoption I would guess this issue may affect a good number of people. I'm happy to take a stab at writing a patch but looking at the code it's somewhat out of my comfort zone and I worry I'd make a hash of it. |
|||
msg255275 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * ![]() |
Date: 2015-11-24 15:51 | |
What is exact version of you Python? I can't identify the line in the traceback. There are few lines "newline=newline, encoding=encoding)" in tempfile.py, but no one is closer to line 753. |
|||
msg255276 - (view) | Author: Anilyka Barry (abarry) * ![]() |
Date: 2015-11-24 16:07 | |
I'm with Serhiy, the line number is inane. Could you put the exact contents of your /usr/lib/python3.5/tempfile.py file somewhere for us to see? Output of sys.version would be nice, too. |
|||
msg255283 - (view) | Author: Hans Lawrenz (Hans Lawrenz) * | Date: 2015-11-24 17:43 | |
Serhiy and Emanuel, I'll paste below the surrounding code and attach the exact tempfile.py. It is the version distributed with the 3.5.0 release. If you take a look at the github repo I linked in the first comment you can also try it out for yourself if you've got vagrant and virtualbox installed. I was able to recreate the error with the provisioning compiling python and with it installing from a ppa. You can see the details in the Vagrantfile. I also had a coworker test in a nearly identical environment and he had the same result. (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type) try: _os.unlink(name) return _io.open(fd, mode, buffering=buffering, newline=newline, encoding=encoding) except: _os.close(fd) raise One final note. I decided to try this out with a windows host and the only clarity this test adds is that the problem doesn't show up there but it fails for its own reasons on all the python versions I tried. See below for reference: Last login: Tue Nov 24 17:16:12 2015 from 10.0.2.2 vagrant@vagrant-ubuntu-trusty-64:~$ cat /vagrant/foo.py import tempfile with tempfile.TemporaryFile(dir="/vagrant") as tf: tf.write("testing testing testing\n".encode('utf-8')) print("Path 2: worked") vagrant@vagrant-ubuntu-trusty-64:~$ python /vagrant/foo.py Traceback (most recent call last): File "/vagrant/foo.py", line 4, in <module> with tempfile.TemporaryFile(dir="/vagrant") as tf: File "/usr/lib/python2.7/tempfile.py", line 495, in TemporaryFile _os.unlink(name) OSError: [Errno 26] Text file busy: '/vagrant/tmpvx7Mie' vagrant@vagrant-ubuntu-trusty-64:~$ python2.7 /vagrant/foo.py Traceback (most recent call last): File "/vagrant/foo.py", line 4, in <module> with tempfile.TemporaryFile(dir="/vagrant") as tf: File "/usr/lib/python2.7/tempfile.py", line 495, in TemporaryFile _os.unlink(name) OSError: [Errno 26] Text file busy: '/vagrant/tmpNXQ6Cf' vagrant@vagrant-ubuntu-trusty-64:~$ python3.4 /vagrant/foo.py Traceback (most recent call last): File "/vagrant/foo.py", line 4, in <module> with tempfile.TemporaryFile(dir="/vagrant") as tf: File "/usr/lib/python3.4/tempfile.py", line 638, in TemporaryFile _os.unlink(name) OSError: [Errno 26] Text file busy: '/vagrant/tmpfwhj7ip4' vagrant@vagrant-ubuntu-trusty-64:~$ python3.5 /vagrant/foo.py Traceback (most recent call last): File "/vagrant/foo.py", line 4, in <module> with tempfile.TemporaryFile(dir="/vagrant") as tf: File "/usr/lib/python3.5/tempfile.py", line 751, in TemporaryFile _os.unlink(name) OSError: [Errno 26] Text file busy: '/vagrant/tmp02s19r_a' vagrant@vagrant-ubuntu-trusty-64:~$ python2.7 --version Python 2.7.6 vagrant@vagrant-ubuntu-trusty-64:~$ python3.4 --version Python 3.4.3 vagrant@vagrant-ubuntu-trusty-64:~$ python3.5 --version Python 3.5.0 |
|||
msg255285 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * ![]() |
Date: 2015-11-24 19:06 | |
Is your file system NFS? |
|||
msg255286 - (view) | Author: Anilyka Barry (abarry) * ![]() |
Date: 2015-11-24 19:38 | |
Your file has a lot of shenanigans that are triggered if 'shutil' fails to be imported, adding an extra 139 lines at the top of the file, which is exactly how offset your traceback is compared to our lines. The rest of the file is virtually identical. This makes me wonder, however, why you have these differences (and that this might not be the only different file). I would still like the output of sys.version, which, unlike 'python --version' contains the build date and time, among other things. Bonus points for the 'sys.implementation' output. It's unlikely to be of relevance, but I would rather explore every possibility. |
|||
msg255287 - (view) | Author: Hans Lawrenz (Hans Lawrenz) * | Date: 2015-11-24 19:40 | |
The file system causing the problem is of type vboxsf which is the Virtualbox shared folder file system type. |
|||
msg255288 - (view) | Author: Hans Lawrenz (Hans Lawrenz) * | Date: 2015-11-24 19:58 | |
Emanuel, sorry, I missed the request for sys.version earlier. The tempfile.py I attached earlier is from the python 3.5 pulled from a ppa. I wouldn't be surprised if it has some patches applied by debian/ubuntu. To be clear though the problem also presents itself with a python 3.5 I compiled. The Vagrantfile in the github repository shows exactly how I compiled it. Here it is from the version from the ppa: Python 3.5.0 (default, Sep 17 2015, 00:00:00) [GCC 4.8.4] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.version '3.5.0 (default, Sep 17 2015, 00:00:00) \n[GCC 4.8.4]' >>> sys.implementation namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-35', hexversion=50659568, name='cpython', version=sys.version_info(major=3, minor=5, micro=0, releaselevel='final', serial=0)) Here are the same for the version I compiled: vagrant@vagrant-ubuntu-trusty-64:~$ python3.5 Python 3.5.0 (default, Nov 24 2015, 19:50:42) [GCC 4.8.4] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.version '3.5.0 (default, Nov 24 2015, 19:50:42) \n[GCC 4.8.4]' >>> sys.implementation namespace(cache_tag='cpython-35', hexversion=50659568, name='cpython', version=sys.version_info(major=3, minor=5, micro=0, releaselevel='final', serial=0)) |
|||
msg255292 - (view) | Author: Martin Panter (martin.panter) * ![]() |
Date: 2015-11-24 22:10 | |
This patch restores the previous behaviour of tolerating fstat() failures other than EBADF. Hans, if you are able to apply it to your compiled version of Python, it might prove that my fstat() theory is correct. |
|||
msg255352 - (view) | Author: Hans Lawrenz (Hans Lawrenz) * | Date: 2015-11-25 13:57 | |
Martin, I applied your patch and it proved your hypothesis. See below for how I tested. I also updated the github repo for others to reproduce if they wish. cd wget https://www.python.org/ftp/python/3.5.0/Python-3.5.0.tar.xz mkdir ~/dist cd ~/dist tar xJf ~/Python-3.5.0.tar.xz cd Python-3.5.0 ./configure --prefix=/home/vagrant/py35/dist && \ make && make install mkdir ~/patch cd ~/patch tar xJf ~/Python-3.5.0.tar.xz cd Python-3.5.0 patch -p1 < /vagrant/fstat-failure.patch ./configure --prefix=/home/vagrant/py35/patch && \ make && make install vagrant@vagrant-ubuntu-trusty-64:~$ ./py35/dist/bin/python3.5 /vagrant/temptest.py Path 1: worked Path 2: ERROR vagrant@vagrant-ubuntu-trusty-64:~$ ./py35/patch/bin/python3.5 /vagrant/temptest.py Path 1: worked Path 2: worked |
|||
msg255553 - (view) | Author: Martin Panter (martin.panter) * ![]() |
Date: 2015-11-28 22:23 | |
Antoine or Bohuslav, since you were involved in Issue 21679, would you be able to comment on fstat-failure.patch, which reverts back to ignoring most fstat() errors? The problem here is that fstat() seems to fail if the directory entry has been unlinked, and the file is on a Virtual Box shared folder filesystem. |
|||
msg255647 - (view) | Author: Bohuslav "Slavek" Kabrda (bkabrda) * | Date: 2015-12-01 11:00 | |
The patch looks good to me. |
|||
msg256001 - (view) | Author: Roundup Robot (python-dev) ![]() |
Date: 2015-12-06 03:51 | |
New changeset 20ea12222b0e by Martin Panter in branch '3.5': Issue #25717: Tolerate fstat() failures in the FileIO constructor https://hg.python.org/cpython/rev/20ea12222b0e New changeset 8c978cbe057c by Martin Panter in branch 'default': Issue #25717: Merge fstat() fix from 3.5 https://hg.python.org/cpython/rev/8c978cbe057c |
|||
msg256002 - (view) | Author: Martin Panter (martin.panter) * ![]() |
Date: 2015-12-06 05:01 | |
Thanks Bohuslav, and also to Hans for helping track this down. |
|||
msg256006 - (view) | Author: STINNER Victor (vstinner) * ![]() |
Date: 2015-12-06 08:33 | |
Martin, please add a comment to explain the rationale on ignoring errors other than EBADF. At least, mention this issue number. |
|||
msg256008 - (view) | Author: Martin Panter (martin.panter) * ![]() |
Date: 2015-12-06 11:15 | |
Okay will do |
|||
msg256009 - (view) | Author: Roundup Robot (python-dev) ![]() |
Date: 2015-12-06 11:20 | |
New changeset e9bf5803b716 by Martin Panter in branch '3.5': Issue #25717: Add comment explaining why errors are ignored https://hg.python.org/cpython/rev/e9bf5803b716 New changeset 8bf69413ec01 by Martin Panter in branch 'default': Issue #25717: Merge comment from 3.5 https://hg.python.org/cpython/rev/8bf69413ec01 |
|||
msg256015 - (view) | Author: STINNER Victor (vstinner) * ![]() |
Date: 2015-12-06 15:30 | |
Thank you. |
|||
msg268783 - (view) | Author: Oleg Babintsev (Oleg Babintsev) | Date: 2016-06-18 09:40 | |
Problem is still actual for Python 2.7 Can you provide the fix for 2.7 too? |
|||
msg268790 - (view) | Author: Martin Panter (martin.panter) * ![]() |
Date: 2016-06-18 09:59 | |
Are you sure Oleg? As far as I understand, Python 2 by default wraps C stdio file objects, and also has Python 3’s file objects in the “io” module. But I expect TemporaryFile() would use the default stdio files, and the cause of this bug, Issue 21679, should only have affected the “io” module in 3.5+. |
|||
msg268792 - (view) | Author: Oleg Babintsev (Oleg Babintsev) | Date: 2016-06-18 10:35 | |
My environment: Windows as host OS, VirtualBox VM, Linux as guest OS When temporary directory is a directory which mounted from the host OS, a "OSError: [Errno 26] Text file busy" exception is thrown. Test file: import tempfile with tempfile.TemporaryFile(dir="/tmp") as tf: tf.write("testing testing testing\n".encode('utf-8')) print("Temp file: worked") "/tmp" directory - is a shared folder (vboxsf) Result: 0e87f2561643">root@0e87f2561643:/# python /etc/odoo/tempfile-test.py Traceback (most recent call last): File "/etc/odoo/tempfile-test.py", line 3, in <module> with tempfile.TemporaryFile(dir="/tmp") as tf: File "/usr/lib/python2.7/tempfile.py", line 513, in TemporaryFile _os.unlink(name) OSError: [Errno 26] Text file busy: '/tmp/tmpsl0JQI' 0e87f2561643">root@0e87f2561643:/# python --version Python 2.7.11+ 0e87f2561643">root@0e87f2561643:/# python Python 2.7.11+ (default, Jun 2 2016, 19:34:15) [GCC 5.3.1 20160528] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.version '2.7.11+ (default, Jun 2 2016, 19:34:15) \n[GCC 5.3.1 20160528]' >>> |
|||
msg268796 - (view) | Author: Martin Panter (martin.panter) * ![]() |
Date: 2016-06-18 11:36 | |
This original bug was about fstat() failing for an anonymous file (after the directory entry was removed). But in your situation the file system refuses to unlink the directory entry. If you think there is something that can be fixed in Python, I suggest open a new bug. But it seems this is working as documented: “Under Unix, the directory entry for the file is removed immediately after the file is created.” In your case, this cannot happen, so you get the error from the OS. |
|||
msg268834 - (view) | Author: Oleg Babintsev (Oleg Babintsev) | Date: 2016-06-18 22:37 | |
Thanks for explanation. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:58:24 | admin | set | github: 69903 |
2016-06-18 22:37:07 | Oleg Babintsev | set | messages: + msg268834 |
2016-06-18 11:36:33 | martin.panter | set | messages:
+ msg268796 versions: + Python 3.5, Python 3.6, - Python 2.7 |
2016-06-18 10:35:39 | Oleg Babintsev | set | messages: + msg268792 |
2016-06-18 09:59:53 | martin.panter | set | messages: + msg268790 |
2016-06-18 09:40:25 | Oleg Babintsev | set | nosy:
+ Oleg Babintsev messages: + msg268783 versions: + Python 2.7, - Python 3.5, Python 3.6 |
2016-02-11 01:43:11 | eryksun | link | issue25639 superseder |
2015-12-06 15:30:20 | vstinner | set | messages: + msg256015 |
2015-12-06 11:20:34 | python-dev | set | messages: + msg256009 |
2015-12-06 11:15:29 | martin.panter | set | messages: + msg256008 |
2015-12-06 08:33:34 | vstinner | set | messages: + msg256006 |
2015-12-06 05:01:41 | martin.panter | set | status: open -> closed resolution: fixed messages: + msg256002 stage: patch review -> resolved |
2015-12-06 03:51:58 | python-dev | set | nosy:
+ python-dev messages: + msg256001 |
2015-12-01 11:00:16 | bkabrda | set | messages: + msg255647 |
2015-11-28 22:23:47 | martin.panter | set | versions:
+ Python 3.6 nosy: + pitrou, bkabrda messages: + msg255553 stage: patch review |
2015-11-25 13:57:02 | Hans Lawrenz | set | messages: + msg255352 |
2015-11-24 22:16:03 | serhiy.storchaka | set | nosy:
+ vstinner |
2015-11-24 22:10:46 | martin.panter | set | files:
+ fstat-failure.patch keywords: + patch messages: + msg255292 |
2015-11-24 19:58:27 | Hans Lawrenz | set | messages: + msg255288 |
2015-11-24 19:40:55 | Hans Lawrenz | set | messages: + msg255287 |
2015-11-24 19:38:21 | abarry | set | messages: + msg255286 |
2015-11-24 19:06:27 | serhiy.storchaka | set | messages: + msg255285 |
2015-11-24 17:43:56 | Hans Lawrenz | set | files:
+ tempfile.py messages: + msg255283 |
2015-11-24 16:07:13 | abarry | set | nosy:
+ abarry messages: + msg255276 |
2015-11-24 15:51:19 | serhiy.storchaka | set | nosy:
+ serhiy.storchaka messages: + msg255275 |
2015-11-24 15:38:38 | Hans Lawrenz | set | messages: + msg255273 |
2015-11-24 08:39:42 | martin.panter | set | messages: + msg255254 |
2015-11-24 06:54:31 | Hans Lawrenz | set | files:
+ tempfile-strace.txt messages: + msg255250 |
2015-11-24 04:42:49 | martin.panter | set | keywords:
+ 3.5regression nosy: + martin.panter messages: + msg255247 |
2015-11-24 04:23:44 | Hans Lawrenz | create |