classification
Title: OpenBSD: socket.recvmsg tests fail with OSError: [Errno 40] Message too long
Type: Stage: resolved
Components: Versions: Python 3.5
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: baikie, brian, ced, giampaolo.rodola, jackdied, janssen, jcea, ncoghlan, neologix, pitrou, python-dev, rosslagerwall, rpointel, synapse, therve, vstinner, wiml, worr
Priority: normal Keywords: patch

Created on 2014-02-18 08:41 by vstinner, last changed 2017-07-03 14:29 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
recvmsg_trunc_emsgsize.patch vstinner, 2014-02-20 23:26 review
recvmsg_linux_freebsd_only.patch worr, 2015-04-18 03:38 review
Messages (15)
msg211493 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-02-18 08:41
http://buildbot.python.org/all/builders/x86%20OpenBSD%205.5%203.x/builds/14/steps/test/logs/stdio

======================================================================
ERROR: testCmsgTrunc1 (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2857, in testCmsgTrunc1
    self.checkTruncatedHeader(self.doRecvmsg(self.serv_sock, len(MSG), 1))
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1872, in doRecvmsg
    result = sock.recvmsg(bufsize, *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTrunc2Int (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2867, in testCmsgTrunc2Int
    SIZEOF_INT * 2))
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1872, in doRecvmsg
    result = sock.recvmsg(bufsize, *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen0 (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2907, in testCmsgTruncLen0
    self.checkTruncatedArray(ancbuf=socket.CMSG_LEN(0), maxdata=0)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2888, in checkTruncatedArray
    len(MSG), ancbuf)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1872, in doRecvmsg
    result = sock.recvmsg(bufsize, *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen0Minus1 (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2874, in testCmsgTruncLen0Minus1
    socket.CMSG_LEN(0) - 1))
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1872, in doRecvmsg
    result = sock.recvmsg(bufsize, *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen0Plus1 (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2913, in testCmsgTruncLen0Plus1
    self.checkTruncatedArray(ancbuf=socket.CMSG_LEN(0) + 1, maxdata=1)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2888, in checkTruncatedArray
    len(MSG), ancbuf)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1872, in doRecvmsg
    result = sock.recvmsg(bufsize, *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen1 (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2920, in testCmsgTruncLen1
    maxdata=SIZEOF_INT)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2888, in checkTruncatedArray
    len(MSG), ancbuf)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1872, in doRecvmsg
    result = sock.recvmsg(bufsize, *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen2Minus1 (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2927, in testCmsgTruncLen2Minus1
    maxdata=(2 * SIZEOF_INT) - 1)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2888, in checkTruncatedArray
    len(MSG), ancbuf)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1872, in doRecvmsg
    result = sock.recvmsg(bufsize, *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testFDPassSeparate (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 258, in _tearDown
    raise exc
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 270, in clientRun
    test_func()
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2722, in _testFDPassSeparate
    array.array("i", [fd1]))]),
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1864, in sendmsgToServer
    *(args + self.sendmsg_to_server_defaults[len(args):]))
OSError: [Errno 22] Invalid argument

======================================================================
ERROR: testFDPassSeparateMinSpace (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 258, in _tearDown
    raise exc
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 270, in clientRun
    test_func()
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2746, in _testFDPassSeparateMinSpace
    array.array("i", [fd1]))]),
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1864, in sendmsgToServer
    *(args + self.sendmsg_to_server_defaults[len(args):]))
OSError: [Errno 22] Invalid argument

======================================================================
ERROR: testCmsgTrunc1 (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2857, in testCmsgTrunc1
    self.checkTruncatedHeader(self.doRecvmsg(self.serv_sock, len(MSG), 1))
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1963, in doRecvmsg
    result = sock.recvmsg_into([buf], *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTrunc2Int (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2867, in testCmsgTrunc2Int
    SIZEOF_INT * 2))
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1963, in doRecvmsg
    result = sock.recvmsg_into([buf], *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen0 (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2907, in testCmsgTruncLen0
    self.checkTruncatedArray(ancbuf=socket.CMSG_LEN(0), maxdata=0)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2888, in checkTruncatedArray
    len(MSG), ancbuf)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1963, in doRecvmsg
    result = sock.recvmsg_into([buf], *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen0Minus1 (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2874, in testCmsgTruncLen0Minus1
    socket.CMSG_LEN(0) - 1))
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1963, in doRecvmsg
    result = sock.recvmsg_into([buf], *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen0Plus1 (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2913, in testCmsgTruncLen0Plus1
    self.checkTruncatedArray(ancbuf=socket.CMSG_LEN(0) + 1, maxdata=1)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2888, in checkTruncatedArray
    len(MSG), ancbuf)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1963, in doRecvmsg
    result = sock.recvmsg_into([buf], *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen1 (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2920, in testCmsgTruncLen1
    maxdata=SIZEOF_INT)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2888, in checkTruncatedArray
    len(MSG), ancbuf)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1963, in doRecvmsg
    result = sock.recvmsg_into([buf], *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen2Minus1 (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2927, in testCmsgTruncLen2Minus1
    maxdata=(2 * SIZEOF_INT) - 1)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2888, in checkTruncatedArray
    len(MSG), ancbuf)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1963, in doRecvmsg
    result = sock.recvmsg_into([buf], *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testFDPassSeparate (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 258, in _tearDown
    raise exc
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 270, in clientRun
    test_func()
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2722, in _testFDPassSeparate
    array.array("i", [fd1]))]),
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1864, in sendmsgToServer
    *(args + self.sendmsg_to_server_defaults[len(args):]))
OSError: [Errno 22] Invalid argument

======================================================================
ERROR: testFDPassSeparateMinSpace (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 258, in _tearDown
    raise exc
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 270, in clientRun
    test_func()
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2746, in _testFDPassSeparateMinSpace
    array.array("i", [fd1]))]),
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1864, in sendmsgToServer
    *(args + self.sendmsg_to_server_defaults[len(args):]))
OSError: [Errno 22] Invalid argument

======================================================================
FAIL: testFDPassSeparate (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2710, in testFDPassSeparate
    maxcmsgs=2)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2637, in checkRecvmsgFDs
    self.assertEqual(msg, MSG)
AssertionError: b'' != b'Michael Gilfix was here\xe1\x88\xb4\r\n'

======================================================================
FAIL: testFDPassSeparateMinSpace (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2734, in testFDPassSeparateMinSpace
    maxcmsgs=2, ignoreflags=socket.MSG_CTRUNC)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2637, in checkRecvmsgFDs
    self.assertEqual(msg, MSG)
AssertionError: b'' != b'Michael Gilfix was here\xe1\x88\xb4\r\n'

======================================================================
FAIL: testFDPassSeparate (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2710, in testFDPassSeparate
    maxcmsgs=2)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2637, in checkRecvmsgFDs
    self.assertEqual(msg, MSG)
AssertionError: b'' != b'Michael Gilfix was here\xe1\x88\xb4\r\n'

======================================================================
FAIL: testFDPassSeparateMinSpace (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2734, in testFDPassSeparateMinSpace
    maxcmsgs=2, ignoreflags=socket.MSG_CTRUNC)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2637, in checkRecvmsgFDs
    self.assertEqual(msg, MSG)
AssertionError: b'' != b'Michael Gilfix was here\xe1\x88\xb4\r\n'
msg211757 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-02-20 22:37
Extract of the POSIX standard for recvmsg():
http://pubs.opengroup.org/onlinepubs/009695399/functions/recvmsg.html

"""
DESCRIPTION

(...) The recvmsg() function shall return the total length of the message. (...) If a message is too long to fit in the supplied buffers, and MSG_PEEK is not set in the flags argument, the excess bytes shall be discarded, and MSG_TRUNC shall be set in the msg_flags member of the msghdr structure. (...)

ERRORS

[EMSGSIZE] The msg_iovlen member of the msghdr structure pointed to by message is less than or equal to 0, or is greater than {IOV_MAX}.
"""

I don't see EMSGSIZE in the Linux manual page of recvmsg(). So it looks like Linux truncates, whereas OpenBSD raises an error. It's probably safe to accept in the unit test that the OS can raise OSError(EMSGSIZE).
msg211762 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-02-20 23:26
recvmsg_trunc_emsgsize.patch: modify recvmsg() tests checking for truncated data to ignore the test on OSError(EMSGSIZE).
msg211767 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-02-20 23:47
For failing "FD pass" tests, see issue #12958: these tests are also skipped on Mac OS X. We should maybe also skip these tests on OpenBSD.
msg211827 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-02-21 08:38
I copy the nosy list from issue #6560 which added the recvmsg() and sendmsg() methods.
msg211834 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-02-21 09:15
I opened the issue #20718 to track the "FD pass" failures.
msg211967 - (view) Author: David Watson (baikie) Date: 2014-02-22 22:23
> I opened the issue #20718 to track the "FD pass" failures.

But the failures in the current issue *all* involve FD passing :)

The "Message too long" errors are in tests where the ancillary data (in this case file descriptors) is truncated, rather than the normal data, so msg_iovlen shouldn't be a problem there (and MSG_CTRUNC would normally be set, rather than MSG_TRUNC).

I suppose the most likely reason for a process to be sent too many file descriptors is that someone is attempting a denial-of-service attack on it, so perhaps returning EMSGSIZE is intended as a "fail safe" response, even if it isn't blessed by POSIX?
msg240918 - (view) Author: William Orr (worr) * Date: 2015-04-14 15:51
In the case of EINVAL, OpenBSD returns this if multiple cmsghdrs are attached to a message with sendmsg(2). In this case, those tests should probably be skipped.

I've attached the patch that combines haypo's patch with skips referencing this issue wrt. multiple cmsg failures
msg241073 - (view) Author: William Orr (worr) * Date: 2015-04-15 02:38
I'm adding another patch, as adding multiple cmsg's to a single message with sendmsg is a Linux-specific extension. This only enables the problematic patches on Linux. This solves the AIX failures in #20718 as well.
msg241092 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-04-15 09:27
Can you please split the changes to skip tests on non-Linux platforms, and changes for OpenBSD/AIX please?
msg241243 - (view) Author: David Watson (baikie) Date: 2015-04-16 17:15
Sending multiple control messages at once isn't Linux-specific - the tests in question passed on FreeBSD.
msg241393 - (view) Author: William Orr (worr) * Date: 2015-04-18 03:38
Double checked on a FreeBSD box and confirmed. I missed the loop in the kernel source.

I've attached a new patch, without the emsgsize change.
msg297120 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-28 01:33
Oh, this issue is now old. Can someone please try the patch? Does it fix the issue?
msg297121 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-28 01:33
Or should we just close the issue as "out of date"?
msg297597 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-03 14:29
I'm sorry but I don't have OpenBSD, and there is no more OpenBSD buildbot, so I close this issue as out of date :-( We failed to find anyone to review the patch since 2 years.
History
Date User Action Args
2017-07-03 14:29:10vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg297597

stage: resolved
2017-06-28 01:33:22vstinnersetmessages: + msg297121
2017-06-28 01:33:09vstinnersetmessages: + msg297120
2015-04-18 03:38:41worrsetfiles: + recvmsg_linux_freebsd_only.patch

messages: + msg241393
2015-04-18 02:55:21worrsetfiles: - recvmsg_trunc_emsgsize_cmsg_linux_only.patch
2015-04-18 02:53:48worrsetfiles: - recvmsg_trunc_emsgsize_cmsg_obsd_skip.patch
2015-04-16 17:15:38baikiesetmessages: + msg241243
2015-04-15 09:27:06vstinnersetmessages: + msg241092
2015-04-15 02:38:22worrsetfiles: + recvmsg_trunc_emsgsize_cmsg_linux_only.patch

messages: + msg241073
2015-04-14 17:17:59schmichaelsetnosy: - schmichael
2015-04-14 15:51:16worrsetfiles: + recvmsg_trunc_emsgsize_cmsg_obsd_skip.patch
versions: + Python 3.5, - Python 3.4
nosy: + worr

messages: + msg240918
2015-04-02 15:57:00Jean-Paul Calderonesetnosy: - exarkun
2015-04-02 15:54:34cedsetnosy: + ced
2014-02-23 00:11:40brett.cannonsetnosy: - brett.cannon
2014-02-22 22:23:11baikiesetmessages: + msg211967
2014-02-21 09:15:59vstinnersetmessages: + msg211834
2014-02-21 08:38:53vstinnersetnosy: + brett.cannon, jcea, exarkun, ncoghlan, janssen, pitrou, therve, jackdied, baikie, giampaolo.rodola, schmichael, synapse, wiml, neologix, rosslagerwall, python-dev, brian
messages: + msg211827
2014-02-20 23:47:54vstinnersetmessages: + msg211767
2014-02-20 23:26:12vstinnersetfiles: + recvmsg_trunc_emsgsize.patch
keywords: + patch
messages: + msg211762
2014-02-20 22:37:25vstinnersetmessages: + msg211757
2014-02-18 08:41:31vstinnercreate