classification
Title: Cleanup io.FileIO
Type: Stage:
Components: Library (Lib) Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: haypo, holdenweb, piotr.dobrogost, python-dev
Priority: normal Keywords: patch

Created on 2015-03-23 16:51 by haypo, last changed 2015-04-06 23:38 by haypo. This issue is now closed.

Files
File name Uploaded Description Edit
fileio_drop_dup_fstat.patch haypo, 2015-03-23 20:46 review
Messages (10)
msg239045 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-23 16:51
While reviewing a Python implementation of io.FileIO (_pyio.FileIO) in the issue #21859, many issues were found in the C implementation. I open an issue to fix them.

We may fix them before or after the Python implementation is merged.
msg239063 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-23 20:46
fileio_drop_dup_fstat.patch: Remove the duplicate call to fstat() in fileio.c. Tests pass, invalid FD are still detected since there is a second call to fstat() which also raises OSError(EBADF) if fstat(fd) fails.
msg239066 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-23 20:54
The previous change related to fstat() is the changeset 3b5279b5bfd1 from the issue #21679. The changeset introduces the private _blksize attribute. The strange thing is that the changelog is:
"Issue #21679: Prevent extraneous fstat() calls during open().  Patch by Bohuslav Kabrda."

fstat() was called twice and it is still called twice. Maybe I missed something.
msg239070 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-23 21:18
See also issue #21861: "io class name are hardcoded in reprs".

In my review of _pyio.FileIO, I asked if it would be possible to use __qualname__ in __getstate__().
msg239548 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-30 01:22
New changeset bc2a22eaa0af by Victor Stinner in branch 'default':
Issue #23752: When built from an existing file descriptor, io.FileIO() now only
https://hg.python.org/cpython/rev/bc2a22eaa0af
msg239577 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-30 08:19
Hum, _Py_fstat() is not used correctly in _io.FilIO: it uses errno instead of GetLastError() to raise the OSError. It's too hard to use the new _Py_fstat() function directly, I modified it to raise directly the exception. So the caller doesn't have to use #ifdef MS_WINDOWS to raise the exception correctly.

I added _Py_fstat_noraise() when it's not possible to use an exception.

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/5978/steps/test/logs/stdio

======================================================================
FAIL: test_fdopen (test.test_os.TestInvalidFD)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_os.py", line 1418, in check
    f(support.make_bad_fd(), *args)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\os.py", line 1028, in fdopen
    return io.open(fd, *args, **kwargs)
OSError: [Errno 0] Error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_os.py", line 1411, in helper
    self.check(getattr(os, f))
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_os.py", line 1420, in check
    self.assertEqual(e.errno, errno.EBADF)
AssertionError: 0 != 9
msg239578 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-30 08:20
New changeset 128f92ae8bae by Victor Stinner in branch 'default':
Issue #23752: _Py_fstat() is now responsible to raise the Python exception
https://hg.python.org/cpython/rev/128f92ae8bae
msg239588 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-30 09:41
The changeset 128f92ae8bae also changed _Py_fstat() to release the GIL when calling fstat(), I forgot to mention it in the changeset.

test_socket now hangs on AMD64 Snow Leop 3.x, I don't know yet if it's related:

http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/2870/steps/test/logs/stdio

[393/393] test_socket
Timeout (1:00:00)!
Thread 0x0000000104fef000 (most recent call first):
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/socket.py", line 288 in _sendfile_use_sendfile
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/test_socket.py", line 5131 in _testOffset
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/test_socket.py", line 278 in clientRun

Thread 0x00007fff71296cc0 (most recent call first):
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/threading.py", line 293 in wait
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/threading.py", line 556 in wait
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/test_socket.py", line 262 in _tearDown
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/case.py", line 580 in run
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/case.py", line 625 in __call__
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/suite.py", line 122 in run
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/suite.py", line 84 in __call__
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/suite.py", line 122 in run
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/suite.py", line 84 in __call__
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/runner.py", line 176 in run
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/support/__init__.py", line 1773 in _run_suite
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/support/__init__.py", line 1807 in run_unittest
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/test_socket.py", line 5340 in test_main
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/regrtest.py", line 1280 in runtest_inner
msg239589 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-30 09:44
Oh, there was another hang before, so it's probably not related:

http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/2852/steps/test/logs/stdio

[393/393] test_ssl
Timeout (1:00:00)!
Thread 0x0000000102d81000 (most recent call first):
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/test_ssl.py", line 2832 in serve
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/threading.py", line 871 in run
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/threading.py", line 923 in _bootstrap_inner
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/threading.py", line 891 in _bootstrap

Thread 0x00007fff71296cc0 (most recent call first):
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/ssl.py", line 628 in do_handshake
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/ssl.py", line 983 in do_handshake
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/ssl.py", line 747 in __init__
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/ssl.py", line 1064 in wrap_socket
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/case.py", line 162 in handle
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/case.py", line 1238 in assertRaisesRegex
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/test_ssl.py", line 2851 in test_handshake_timeout
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/case.py", line 577 in run
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/case.py", line 625 in __call__
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/suite.py", line 122 in run
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/suite.py", line 84 in __call__
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/suite.py", line 122 in run
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/suite.py", line 84 in __call__
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/unittest/runner.py", line 176 in run
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/support/__init__.py", line 1773 in _run_suite
msg240193 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-04-06 23:38
Buildbot issues were unrelated and have been fixed. I close the issue.
History
Date User Action Args
2015-04-06 23:38:35hayposetstatus: open -> closed
resolution: fixed
messages: + msg240193
2015-03-30 09:44:58hayposetmessages: + msg239589
2015-03-30 09:41:18hayposetmessages: + msg239588
2015-03-30 08:20:08python-devsetmessages: + msg239578
2015-03-30 08:19:58hayposetstatus: closed -> open

nosy: + holdenweb
messages: + msg239577

resolution: fixed -> (no value)
2015-03-30 01:23:26hayposetstatus: open -> closed
resolution: fixed
2015-03-30 01:22:53python-devsetnosy: + python-dev
messages: + msg239548
2015-03-23 22:57:03piotr.dobrogostsetnosy: + piotr.dobrogost
2015-03-23 21:18:52hayposetmessages: + msg239070
2015-03-23 20:54:21hayposetmessages: + msg239066
2015-03-23 20:46:59hayposetfiles: + fileio_drop_dup_fstat.patch
keywords: + patch
messages: + msg239063
2015-03-23 16:51:05haypocreate