The fix seems to be simple: skip the test if write() fails with errno.ENOSPC: "OSError: [Errno 28] No space left on device".


AMD64 Windows7 SP1 3.x:
https://buildbot.python.org/all/#/builders/40/builds/2628

======================================================================
ERROR: test_large_file_ops (test.test_io.CIOTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_io.py", line 608, in test_large_file_ops
    self.large_file_ops(f)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_io.py", line 389, in large_file_ops
    self.assertEqual(f.write(b"xxx"), 3)
OSError: [Errno 28] No space left on device

======================================================================
ERROR: test_large_file_ops (test.test_io.PyIOTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_io.py", line 608, in test_large_file_ops
    self.large_file_ops(f)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_io.py", line 389, in large_file_ops
    self.assertEqual(f.write(b"xxx"), 3)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\_pyio.py", line 1698, in write
    return os.write(self._fd, b)
OSError: [Errno 28] No space left on device


No idea if the following test_asyncio failure is related.

======================================================================
ERROR: test_huge_content (test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_asyncio\test_sock_lowlevel.py", line 170, in test_huge_content
    self.loop.run_until_complete(
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\base_events.py", line 608, in run_until_complete
    return future.result()
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_asyncio\test_sock_lowlevel.py", line 157, in _basetest_huge_content
    data = await self.loop.sock_recv(sock, DATA_SIZE)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\selector_events.py", line 362, in sock_recv
    return await fut
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\selector_events.py", line 373, in _sock_recv
    data = sock.recv(n)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

Hi,

I'd like to claim this task if nobody else already started working on it.

To check - do we definitely want to just skip the test if the below OSError is thrown? Could we attempt to clean wherever the test is writing to before the test is run/figure out what is using all the space?

> To check - do we definitely want to just skip the test if the below OSError is thrown?

Note: I don't propose to ignore any OSError, only if exc.errno == errno.ENOSPC.

> Could we attempt to clean wherever the test is writing to before the test is run/figure out what is using all the space?

Maybe some tests can be enhanced, but that's not directly related to this issue.

In general, Python tests don't leak any resource: memory, files, etc. If there is a leak, it must be fixed.

The test creates a file which can take up to 2 GiB on the disk. It's a big file. I don't think that other tests leak such huge file.

The test requires the "largefile" resource on Windows. If "-u largefile" is specified and the test is failed, it is a correct behavior.

The correct solution is either increase the amount of available free disk space or remove the "largefile" resource on that buildbot.

libregrtest usage says:

    largefile - It is okay to run some test that may create huge
                files.  These tests can take a long time and may
                consume >2 GiB of disk space temporarily.

> The test requires the "largefile" resource on Windows. If "-u largefile" is specified and the test is failed, it is a correct behavior.

Ok, but it's a practical issue. How can a buildbot worker maintainer ensure that a machine always has 2 GiB of free disk space? Is it really a Python bug if the machine has lower than 2 GiB of free space?

The test suite fails if there is less than 500 or 600 MB of RAM, not counting bigmem tests. It is not a Python bug, but an environment issue. A buildbot worker should have some free disk space and RAM, and set tests running options correspondingly.

Serhiy:
> The test suite fails if there is less than 500 or 600 MB of RAM, not counting bigmem tests. It is not a Python bug, but an environment issue. A buildbot worker should have some free disk space and RAM, and set tests running options correspondingly.

How do you plan to fix buildbots? Contact each buildbot owner and asks them to ensure that largefile is only used with at least 2 GiB of free space? And maybe install monitor to get an alarm if it's no longer the case?

Maybe some buildbot owners will just remove largefile, which would not be the expected reaction, no?

Yes, it is a solution.

In this particular case it may be that the disk is fulled by test files leaked from the past test runs.

Serhiy, Victor:

I didn't get what's your suggested solution for this task.

Should we just close it as "wontfix", can I go ahead and merge the existing PR, or do you have a better suggestion on how to solve this issue?

According to Serhiy, the right solution is to contact buildbot owners to ask them to ensure that all workers have enough free disk space: at least 2 GiB.