classification
Title: Possible integer overflow in iov_setup()
Type: crash Stage: resolved
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: miss-islington, ned.deily, pablogsal, ronaldoussoren, serhiy.storchaka
Priority: high Keywords: patch

Created on 2018-06-15 18:14 by serhiy.storchaka, last changed 2018-09-22 18:31 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 7931 merged serhiy.storchaka, 2018-06-26 16:46
PR 8583 merged miss-islington, 2018-07-31 07:25
PR 8584 merged serhiy.storchaka, 2018-07-31 07:38
Messages (8)
msg319636 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-06-15 18:14
The iov_setup() helper in posixmodule.c returns the total size of all buffers. But there is possible an integer overflow because the sequence of buffers can contain the same buffer repeated multiple times.

On 32-bit platform:

>>> import os
>>> f = open('/tmp/temp', 'wb')
>>> os.writev(f.fileno(), [b'x' * 2**16] * 2**15)
-1

Since the overflowed sum is negative, os_writev_impl() returns -1 as a signal of error, but since the exception is not set, -1 is returned as the result of os.writev(). If the overflowed sum is not negative, the sequence of buffers is passed to OS and an OSError is raised:

>>> os.writev(f.fileno(), [b'x' * 2**16] * 2**16)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 22] Invalid argument

I have not tested (because have not installed corresponding 32-bit OSes, and it is harder to reproduce on 64-bit), but seems this can even cause a crash in os.sendfile() on FreeBSD, DragonFly BSD and Mac OS.

This sum is used only in os.sendfile() on Mac OS. In all other cases it is enough to return just an error flag. I can't find the documentation for os.sendfile() on Mac OS for checking if this value actually is needed.
msg319641 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-06-15 18:42
I can reproduce the issue on a i686 GNU/Linux  Debian system:

>>> import os
>>> f = open('/tmp/temp', 'wb')
>>> l = os.writev(f.fileno(), [b'x' * 2**16] * 2**15)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: error return without exception set
msg319645 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-06-15 18:47
Docs for sendfile in macOS:

https://www.unix.com/man-page/osx/2/sendfile/
msg319757 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-16 16:02
Test case:

import os
fo = open('/tmp/temp', 'wb')
fi = open('/tmp/temp', 'rb')
os.sendfile(fo.fileno(), fi.fileno(), 0, 0, headers=[b'x' * 2**16] * 2**15)

--
run against current master HEAD (2f9cbaa8b2190b6dfd3157ede9b6973523a3b939, as of 2018-06-15)
--with-pydebug
current macOS 10.13.5


64-bit Python

$ ./bin/python3.8 ~/Desktop/test_s.py
Traceback (most recent call last):
 File "/Users/nad/Desktop/test_s.py", line 4, in <module>
   os.sendfile(fo.fileno(), fi.fileno(), 0, 0, headers=[b'x' * 2**16] * 2**15)
OSError: [Errno 38] Socket operation on non-socket
sys:1: ResourceWarning: unclosed file <_io.BufferedWriter name='/tmp/temp'>
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name='/tmp/temp'>


32-bit Python

$ ./bin/python3.8-32 ~/Desktop/test_s.py
Fatal Python error: a function returned NULL without setting an error
SystemError: <built-in function sendfile> returned NULL without setting an error

Current thread 0xa983a1c0 (most recent call first):
 File "/Users/nad/Desktop/test_s.py", line 4 in <module>
Abort trap: 6
msg319879 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-06-18 12:56
Thank you Pablo and Ned!

Seems there is other bug on Mac OS, not related to integer overflow. I am working on it.
msg322739 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-31 07:24
New changeset 9d5727326af53ddd91016d98e16ae7cf829caa95 by Serhiy Storchaka in branch 'master':
bpo-33871: Fix os.sendfile(), os.writev(), os.readv(), etc. (GH-7931)
https://github.com/python/cpython/commit/9d5727326af53ddd91016d98e16ae7cf829caa95
msg322746 - (view) Author: miss-islington (miss-islington) Date: 2018-07-31 09:20
New changeset 3e4b68875917a4605b45918f9e3232730fed9399 by Miss Islington (bot) in branch '3.7':
bpo-33871: Fix os.sendfile(), os.writev(), os.readv(), etc. (GH-7931)
https://github.com/python/cpython/commit/3e4b68875917a4605b45918f9e3232730fed9399
msg322749 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-31 09:59
New changeset ada5d99306dc8af21c32cefb3d86891e8553dbc6 by Serhiy Storchaka in branch '3.6':
[3.6] bpo-33871: Fix os.sendfile(), os.writev(), os.readv(), etc. (GH-7931) (GH-8584)
https://github.com/python/cpython/commit/ada5d99306dc8af21c32cefb3d86891e8553dbc6
History
Date User Action Args
2018-09-22 18:31:42serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-07-31 09:59:03serhiy.storchakasetmessages: + msg322749
2018-07-31 09:20:13miss-islingtonsetnosy: + miss-islington
messages: + msg322746
2018-07-31 07:38:38serhiy.storchakasetpull_requests: + pull_request8093
2018-07-31 07:25:13miss-islingtonsetpull_requests: + pull_request8092
2018-07-31 07:24:57serhiy.storchakasetmessages: + msg322739
2018-06-26 16:46:56serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request7538
2018-06-18 12:56:32serhiy.storchakasetpriority: normal -> high

messages: + msg319879
2018-06-16 16:02:48ned.deilysetmessages: + msg319757
2018-06-15 18:47:20pablogsalsetmessages: + msg319645
2018-06-15 18:42:22pablogsalsetnosy: + pablogsal
messages: + msg319641
2018-06-15 18:14:36serhiy.storchakacreate