This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author vstinner
Recipients giampaolo.rodola, nanjekyejoannah, pablogsal, serhiy.storchaka, vstinner
Date 2019-09-30.07:37:51
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <>
Error on a 32-bit buildbot worker where ssize_t maximum = 2,147,483,647 (2**31-1) bytes = ~2.0 GiB.

test_largefile uses:

# size of file to create (>2 GiB; 2 GiB == 2,147,483,648 bytes)
size = 2_500_000_000

x86 Gentoo Installed with X 3.x:

ERROR: test_it (test.test_largefile.TestCopyfile)
Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.9/test/", line 160, in test_it
    shutil.copyfile(TESTFN, TESTFN2)
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.9/", line 266, in copyfile
    _fastcopy_sendfile(fsrc, fdst)
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.9/", line 145, in _fastcopy_sendfile
    sent = os.sendfile(outfd, infd, offset, blocksize)
OverflowError: Python int too large to convert to C ssize_t

On Linux (Fedora 30), man sendfile shows me the prototype:

       ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count);

Extract of Lib/

    # Hopefully the whole file will be copied in a single call.
    # sendfile() is called in a loop 'till EOF is reached (0 return)
    # so a bufsize smaller or bigger than the actual file size
    # should not make any difference, also in case the file content
    # changes while being copied.
        blocksize = max(os.fstat(infd).st_size, 2 ** 23)  # min 8MB
    except Exception:
        blocksize = 2 ** 27  # 128MB

    offset = 0
    while True:
            sent = os.sendfile(outfd, infd, offset, blocksize)
        except OSError as err:
            if sent == 0:
                break  # EOF
            offset += sent

Extract of the Linux implementation of os.sendfile():

    int in, out;
    Py_ssize_t ret;
    off_t offset;
    Py_ssize_t count;
    PyObject *offobj;
    static char *keywords[] = {"out", "in",
                                "offset", "count", NULL};
    if (!PyArg_ParseTupleAndKeywords(args, kwdict, "iiOn:sendfile",
            keywords, &out, &in, &offobj, &count))
        return NULL;
    if (!Py_off_t_converter(offobj, &offset))
        return NULL;

    do {
        ret = sendfile(out, in, &offset, count);
    } while (ret < 0 && errno == EINTR && !(async_err = PyErr_CheckSignals()));


static int
Py_off_t_converter(PyObject *arg, void *addr)
    *((Py_off_t *)addr) = PyLong_AsLongLong(arg);
    *((Py_off_t *)addr) = PyLong_AsLong(arg);
    if (PyErr_Occurred())
        return 0;
    return 1;

I understand that the error comes from the 4th sendfile() parameter: "count". The C code (of the Linux implementation) uses the "n" format for Py_ssize_t: Python/getargs.c calls PyLong_AsSsize_t().

On a 64-bit system, it's less likely to reach Py_ssize_t maximum value (max = 2**63-1), but it's easy to reach on a 32-bit system (max = 2**31-1).
Date User Action Args
2019-09-30 07:37:52vstinnersetrecipients: + vstinner, giampaolo.rodola, serhiy.storchaka, pablogsal, nanjekyejoannah
2019-09-30 07:37:51vstinnersetmessageid: <>
2019-09-30 07:37:51vstinnerlinkissue38319 messages
2019-09-30 07:37:51vstinnercreate