Message109001
> When talking about pickle speed performance, we need to take into
> account that most use cases for pickle are i/o bound. IOW, the time
> to pickle and unpickle is dominated by the time needed to read/write
> the pickle to/from disk or to send it across the wire. In those
> common use cases, any time spent compressing/decompressing is more
> than made for by savings in transport/storage time.
This is an oversimplification. Transport and storage performance is
mostly orthogonal to CPU performance; depending on the situation, you
might care about the former, or the latter, or both. Adding a
compression layer obviously taxes the CPU more, while optimizing
bytecode decreases CPU consumption as I've pointed out.
(for example, it is quite unlikely, IMO, that ZODB setups are I/O bound
rather than CPU bound, given that it's a database system written in
Python; and even optimized databases such as PostgreSQL can exhibit
CPU-bound behaviour, depending on the hardware configuration, the
database size and the queries issued)
> This is a fairly common speed optimization and it would be nice if
> pickles supported it directly (i.e. the sender specifies a compression
> option and the receiver automatically detects the option from an
> opcode in the pickle.
I'm not sure what would be the point. It is trivially easy for people to
add that compression layer to their own protocols if they want to. Also,
the C code for pickle doesn't have to be complicated by interfacing it
with various third-party compression libraries.
(a doc addition would be more reasonable) |
|
Date |
User |
Action |
Args |
2010-06-30 17:54:45 | pitrou | set | recipients:
+ pitrou, fdrake, rhettinger, mark.dickinson, belopolsky, alexandre.vassalotti |
2010-06-30 17:54:43 | pitrou | link | issue9120 messages |
2010-06-30 17:54:42 | pitrou | create | |
|