Title: `OverflowError: signed integer is greater than maximum` in for files larger than 2GB
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.10, Python 3.9, Python 3.8
Status: open Resolution:
Dependencies: Superseder:
Assigned To: methane Nosy List: amacd31, christian.heimes, methane, ronaldoussoren
Priority: normal Keywords:

Created on 2021-01-07 08:03 by amacd31, last changed 2021-01-08 13:05 by christian.heimes.

Messages (4)
msg384565 - (view) Author: Andrew MacDonald (amacd31) Date: 2021-01-07 08:03
When attempting to read a large file (> 2GB) over HTTPS the read fails with "OverflowError: signed integer is greater than maximum".

This occurs with Python >=3.8 and I've been able to reproduce the problem with the below snippet of code on Linux, Mac OS X, and Windows (the remote file can be any HTTPS hosted file larger than 2GB, e.g. an empty file generated with `dd if=/dev/zero of=2g.img bs=1 count=0 seek=2G` will also do the job.).

import http.client
connection = http.client.HTTPSConnection("")
connection.request("GET", "/pub/centos/8/isos/x86_64/CentOS-8.3.2011-x86_64-dvd1.iso")
response = connection.getresponse()
data =

Doing a git bisect it looks like this is the result of a change in commit d6bf6f2d0c83f0c64ce86e7b9340278627798090 ( Looking over the associated issue and commit message it seems like this was not an intended outcome for the change.
msg384566 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-01-07 08:15
I cannot lift the overflow restriction until we drop support for OpenSSL 1.0.2. The function SSL_write() and SSL_read() are limited to signed 32bit int. OpenSSL 1.1.1 has new SSL_write_ex() and SSL_read_ex() functions that support size_t. Even size_t limits the maximum value to unsigned 32bit (~4GB) on 32bit systems.
msg384649 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2021-01-08 09:52
The API documentation already implies that write might not write the entire buffer because it returns the number of bytes actually written (just like os.write).  

A possible workaround on the SSL layer is hence to clamp the amount of bytes to write to MAX_INT (or later MAX_SSIZE_T) bytes. 

That said, this does require checking that users of the SSL layer write method in the stdib actually check for the number of bytes written, otherwise we'd exchange the exception to a silent error.
msg384658 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-01-08 13:05
That's a good idea, Ronald! socket.c:sock_send_impl() already clamps the input length on Windows:

    if (ctx->len > INT_MAX)
        ctx->len = INT_MAX;
    ctx->result = send(s->sock_fd, ctx->buf, (int)ctx->len, ctx->flags);
    ctx->result = send(s->sock_fd, ctx->buf, ctx->len, ctx->flags);

I could implement a similar logic for SSLSocket. Applications have to check the return value of send() any way or use sendall(). The socket.send() method / send(2) libc function may also write less bytes.
Date User Action Args
2021-01-08 13:05:25christian.heimessetmessages: + msg384658
2021-01-08 09:52:17ronaldoussorensetnosy: + ronaldoussoren
messages: + msg384649
2021-01-07 08:15:02christian.heimessetmessages: + msg384566
2021-01-07 08:07:32christian.heimessetassignee: christian.heimes -> methane

components: + Library (Lib), - SSL
nosy: + methane
2021-01-07 08:03:09amacd31create