Author serhiy.storchaka
Recipients THRlWiTi, christian.heimes, ezio.melotti, gregory.p.smith, inada.naoki, martin.panter, n, serhiy.storchaka, terry.reedy, vstinner
Date 2017-01-04.12:27:45
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1483532865.54.0.787647906852.issue17301@psf.upfronthosting.co.za>
In-reply-to
Content
Python interpreter itself takes 20-30 MB of memory. Using it on a machine that has no few tens of free memory doesn't make much sense. MicroPython can be an exception, but it has special modules for low-level programming.

If you need to proceed so large bytearray that creating yet one copy of it is not possible, it can be translated by separate elements or smaller chunks.

$ ./python -m perf timeit -s "table = bytes(range(256)).swapcase(); data = bytearray(range(256))*1000" -- "for i in range(len(data)): data[i] = table[data[i]]"
Median +- std dev: 205 ms +- 10 ms
$ ./python -m perf timeit -s "table = bytes(range(256)).swapcase(); data = bytearray(range(256))*1000" -- "for i in range(0, len(data), 100): data[i:i+100] = data[i:i+100].translate(table)"
Median +- std dev: 12.9 ms +- 0.6 ms
$ ./python -m perf timeit -s "table = bytes(range(256)).swapcase(); data = bytearray(range(256))*1000" -- "for i in range(0, len(data), 1000): data[i:i+1000] = data[i:i+1000].translate(table)"
Median +- std dev: 3.12 ms +- 0.22 ms
$ ./python -m perf timeit -s "table = bytes(range(256)).swapcase(); data = bytearray(range(256))*1000" -- "for i in range(0, len(data), 10000): data[i:i+10000] = data[i:i+10000].translate(table)"
Median +- std dev: 1.79 ms +- 0.14 ms

Translating by chunks also works with data that can't be fit in memory at all. You can mmap large file or read and write it by blocks. In-place bytearray method wouldn't help in this case.
History
Date User Action Args
2017-01-04 12:27:45serhiy.storchakasetrecipients: + serhiy.storchaka, terry.reedy, gregory.p.smith, vstinner, christian.heimes, ezio.melotti, inada.naoki, THRlWiTi, martin.panter, n
2017-01-04 12:27:45serhiy.storchakasetmessageid: <1483532865.54.0.787647906852.issue17301@psf.upfronthosting.co.za>
2017-01-04 12:27:45serhiy.storchakalinkissue17301 messages
2017-01-04 12:27:45serhiy.storchakacreate