This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Expose the `length` arg from shutil.copyfileobj for public use
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: goodboy, methane, rhettinger, serhiy.storchaka, terry.reedy
Priority: normal Keywords:

Created on 2017-02-27 00:57 by goodboy, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 328 closed goodboy, 2017-02-27 01:19
Messages (5)
msg288616 - (view) Author: Tyler Goodlet (goodboy) * Date: 2017-02-27 00:57
It would be handy to be able to pass the `length` default argument defined by `shutil.copyfileobj()` to other public functions througout the module. When copying very large files (1GB +), increasing the memory buffer can divide the copy time in half and currently the only way to accomplish this is by overriding the needed function(s) which call `shutil.copyfileobj()`.

I propose a simple non-invasive change where the following functions will also expose the `length` kwarg and pass it downwards to `copyfileobj`:
- copyfile
- copy
- copy2
msg288962 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-03-04 05:26
This seems reasonable to me.
msg288974 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2017-03-04 09:06
How about increasing default value to 32KiB or 64KiB too?

binutils's cp works well for most cases, while it doesn't have option to specify blocksize.
I want Python's copy functions works nice enough for common cases too.

binutils cp uses 128KiB block for normal file. see more information in https://github.com/coreutils/coreutils/blob/master/src/ioblksize.h

But copyfileobj can be used other file-likes.
On Linux, typical socket buffer size is 128KiB, and typical pipe buffer size is 64KiB.  So block size larger than 64KiB may cause unneeded blocking.
That's why I suggested 32KiB or 64KiB.
msg305625 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-06 10:11
Could you please provide any benchmarks Tyler?
msg336504 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2019-02-25 09:27
bpo-33671 fixed this.
History
Date User Action Args
2022-04-11 14:58:43adminsetgithub: 73845
2019-02-25 09:27:42methanesetstatus: open -> closed
versions: + Python 3.8, - Python 3.7
messages: + msg336504

resolution: fixed
stage: test needed -> resolved
2017-11-06 10:11:21serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg305625
2017-03-04 09:06:50methanesetnosy: + methane
messages: + msg288974
2017-03-04 05:26:00rhettingersetnosy: + rhettinger
messages: + msg288962
2017-03-03 21:55:28terry.reedysetnosy: + terry.reedy

type: enhancement
stage: test needed
2017-02-27 19:16:23Mariattasetcomponents: + Library (Lib)
2017-02-27 01:19:32goodboysetpull_requests: + pull_request288
2017-02-27 00:57:24goodboycreate