Author graham.coster
Recipients James Hennessy, graham.coster, inada.naoki, martin.panter, r.david.murray, serhiy.storchaka
Date 2019-11-26.02:00:41
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <B991E2C3-55F8-458A-AA34-6FBC28108FC1@yahoo.co.uk>
In-reply-to <23A63731-1232-4D0F-A8C7-9FAA73A7A814@yahoo.co.uk>
Content
This may be a silly question, however, does SpooledTemporaryFile need to exist at all?

From some testing on macOS, SpooledTemporaryFile appeared to never have a performance advantage over OS file caching, but with max_size greater than 4GB, it was a significant disadvantage. So, if the purpose of SpooledTemporaryFile is to increase performance, it may not work.

I found that the macOS built-in file cache was increasing in size as I wrote bigger TemporaryFile files, up to some limit the OS had decided. So, it seems the OS is automatically doing the same job as SpooledTemporaryFile.  Once the OS decided to write to disk, there was no sudden hit to performance, it just slowed down.

However, when SpooledTemporaryFile rolled-over large max_size files, there was a temporary big hit to performance, which then became a consistent slow down the same as a TemporaryFile that had exceeded the OS file cache.

A big issue came with very large SpooledTemporaryFile  max_sizes hogging RAM and causing the OS to start swapping all processes.  This caused a huge performance hit to my program and the system as a whole. Once my program did finish, it took the system considerable time to reclaim swap.

I’m guessing SpooledTemporaryFile may have benefits on light weight embedded OSes that have no, or poor, file caching.  However, tuning the max_size to work with embedded systems’ limited RAM could be tricky for developers and would be hardware dependent. So, perhaps leaving file caching to the underlying operating systems is actually a better, and safer, option than offering it in Python?

If there are no benefits to SpooledTemporaryFile, should it be deprecated? If so, as it is phasesd out, could it be patched to be a TemporaryFile wrapper, with no rollover functionality?
History
Date User Action Args
2019-11-26 02:00:42graham.costersetrecipients: + graham.coster, r.david.murray, inada.naoki, martin.panter, serhiy.storchaka, James Hennessy
2019-11-26 02:00:42graham.costerlinkissue26730 messages
2019-11-26 02:00:41graham.costercreate