classification
Title: [EASY] test_io: test_large_file_ops() failed on AMD64 Windows7 SP1 3.x with: [Errno 28] No space left on device
Type: Stage: resolved
Components: Tests Versions: Python 3.9, Python 3.8, Python 3.7, Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Ali McMaster, jkloth, potomak, serhiy.storchaka, vstinner
Priority: normal Keywords: easy, patch

Created on 2019-06-24 11:44 by vstinner, last changed 2019-07-29 23:38 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 14356 closed potomak, 2019-06-24 23:53
Messages (11)
msg346383 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-24 11:44
The fix seems to be simple: skip the test if write() fails with errno.ENOSPC: "OSError: [Errno 28] No space left on device".


AMD64 Windows7 SP1 3.x:
https://buildbot.python.org/all/#/builders/40/builds/2628

======================================================================
ERROR: test_large_file_ops (test.test_io.CIOTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_io.py", line 608, in test_large_file_ops
    self.large_file_ops(f)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_io.py", line 389, in large_file_ops
    self.assertEqual(f.write(b"xxx"), 3)
OSError: [Errno 28] No space left on device

======================================================================
ERROR: test_large_file_ops (test.test_io.PyIOTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_io.py", line 608, in test_large_file_ops
    self.large_file_ops(f)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_io.py", line 389, in large_file_ops
    self.assertEqual(f.write(b"xxx"), 3)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\_pyio.py", line 1698, in write
    return os.write(self._fd, b)
OSError: [Errno 28] No space left on device


No idea if the following test_asyncio failure is related.

======================================================================
ERROR: test_huge_content (test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_asyncio\test_sock_lowlevel.py", line 170, in test_huge_content
    self.loop.run_until_complete(
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\base_events.py", line 608, in run_until_complete
    return future.result()
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_asyncio\test_sock_lowlevel.py", line 157, in _basetest_huge_content
    data = await self.loop.sock_recv(sock, DATA_SIZE)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\selector_events.py", line 362, in sock_recv
    return await fut
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\selector_events.py", line 373, in _sock_recv
    data = sock.recv(n)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
msg346443 - (view) Author: Giovanni Cappellotto (potomak) * Date: 2019-06-24 22:04
Hi,

I'd like to claim this task if nobody else already started working on it.
msg346450 - (view) Author: Ali McMaster (Ali McMaster) Date: 2019-06-24 23:28
To check - do we definitely want to just skip the test if the below OSError is thrown? Could we attempt to clean wherever the test is writing to before the test is run/figure out what is using all the space?
msg346467 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-25 01:08
> To check - do we definitely want to just skip the test if the below OSError is thrown?

Note: I don't propose to ignore any OSError, only if exc.errno == errno.ENOSPC.

> Could we attempt to clean wherever the test is writing to before the test is run/figure out what is using all the space?

Maybe some tests can be enhanced, but that's not directly related to this issue.

In general, Python tests don't leak any resource: memory, files, etc. If there is a leak, it must be fixed.

The test creates a file which can take up to 2 GiB on the disk. It's a big file. I don't think that other tests leak such huge file.
msg346614 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-06-26 13:15
The test requires the "largefile" resource on Windows. If "-u largefile" is specified and the test is failed, it is a correct behavior.

The correct solution is either increase the amount of available free disk space or remove the "largefile" resource on that buildbot.
msg346616 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-26 13:36
libregrtest usage says:

    largefile - It is okay to run some test that may create huge
                files.  These tests can take a long time and may
                consume >2 GiB of disk space temporarily.

> The test requires the "largefile" resource on Windows. If "-u largefile" is specified and the test is failed, it is a correct behavior.

Ok, but it's a practical issue. How can a buildbot worker maintainer ensure that a machine always has 2 GiB of free disk space? Is it really a Python bug if the machine has lower than 2 GiB of free space?
msg346641 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-06-26 15:52
The test suite fails if there is less than 500 or 600 MB of RAM, not counting bigmem tests. It is not a Python bug, but an environment issue. A buildbot worker should have some free disk space and RAM, and set tests running options correspondingly.
msg346642 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-26 15:56
Serhiy:
> The test suite fails if there is less than 500 or 600 MB of RAM, not counting bigmem tests. It is not a Python bug, but an environment issue. A buildbot worker should have some free disk space and RAM, and set tests running options correspondingly.

How do you plan to fix buildbots? Contact each buildbot owner and asks them to ensure that largefile is only used with at least 2 GiB of free space? And maybe install monitor to get an alarm if it's no longer the case?

Maybe some buildbot owners will just remove largefile, which would not be the expected reaction, no?
msg346645 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-06-26 16:01
Yes, it is a solution.

In this particular case it may be that the disk is fulled by test files leaked from the past test runs.
msg346698 - (view) Author: Giovanni Cappellotto (potomak) * Date: 2019-06-26 23:34
Serhiy, Victor:

I didn't get what's your suggested solution for this task.

Should we just close it as "wontfix", can I go ahead and merge the existing PR, or do you have a better suggestion on how to solve this issue?
msg348699 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-29 23:38
According to Serhiy, the right solution is to contact buildbot owners to ask them to ensure that all workers have enough free disk space: at least 2 GiB.
History
Date User Action Args
2019-07-29 23:38:01vstinnersetstatus: open -> closed
resolution: not a bug
messages: + msg348699

stage: patch review -> resolved
2019-06-26 23:34:15potomaksetmessages: + msg346698
2019-06-26 16:01:21serhiy.storchakasetmessages: + msg346645
2019-06-26 15:56:05vstinnersetmessages: + msg346642
2019-06-26 15:52:05serhiy.storchakasetmessages: + msg346641
2019-06-26 13:36:16vstinnersetmessages: + msg346616
2019-06-26 13:15:09serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg346614
2019-06-25 01:08:47vstinnersetmessages: + msg346467
2019-06-24 23:53:08potomaksetkeywords: + patch
stage: patch review
pull_requests: + pull_request14173
2019-06-24 23:28:54Ali McMastersetnosy: + Ali McMaster
messages: + msg346450
2019-06-24 22:04:32potomaksetnosy: + potomak
messages: + msg346443
2019-06-24 16:14:58jklothsetnosy: + jkloth
2019-06-24 11:45:11vstinnersetcomponents: + Tests
versions: + Python 2.7, Python 3.7, Python 3.8, Python 3.9
2019-06-24 11:44:29vstinnercreate