New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
streaming struct unpacking #62004
Comments
For certain applications, you want to unpack repeatedly the same pattern. This came in bpo-17618 (base85 decoding), where you want to unpack a stream of bytes as 32-bit big-endian unsigned ints. The solution adopted in bpo-17618 patch (struct.Struct("!{}I")) is clearly suboptimal. I would suggest something like a iter_unpack() function which would repeatedly yield tuples until the bytes object is over. |
(my initial intuition here was to use memoryview.cast() but it doesn't support non-native formats) |
Here is a patch (still lacking docs). Comments welcome. |
Perhaps we need not iter_unpack(), but a grouper (some sort of)? def grouper(seq, size):
for i in range(0, len(seq), size):
yield seq[i: i + size]
unpack = struct.Struct('!I').unpack
for chunk in grouper(data, 4):
word, = unpack(chunk)
... |
Well, according to a quick benchmark, iter_unpack() is 3x to 6x faster than the grouper() + unpack() recipe. |
For the record, here is the benchmark script. |
I like the idea of this. Two comments:
|
On 20 Apr 2013, at 23:01, Martin Morrison <report@bugs.python.org> wrote:
Thinking about this more, the functionality is probably too radically different to overload the same function, so I withdraw the suggestion. |
Yes, good catch.
We generally consider it bad API design when a parameter changes the |
This seems like an attractive idea. There's definitely a need for repeated unpacking with the same pattern, and I agree that putting the repetition into the pattern is suboptimal (not least from the point of view of caching structs). One thing that feels a bit unorthogonal is that this is doing two things at once: both allowing for repetition of a pattern, and also adding the lazy iteration. I'd guess that there's also a use-case for allowing repetition but not returning an iterator; but then that's easily covered by list(iter_unpack). +1 from me. Hmm; the name. 'iterunpack'? 'iter_unpack'? 'unpack_stream'? 'unpack_all'? Would we want something similar for packing, too? I guess that's effectively covered by b''.join(s.pack(item) for item in ...). |
Yes, It's mainly because a grouper written on Python. When it will be implemented in C, the difference will be less. This function will be useful beside struct. Note that in my patch for bpo-17618 struct.Struct("!{}I") is not used. As for extending Struct, what you think about a more powerful feature? About a method which returns not an iterator, but an iterable and indexable sequence. Here is a sample Python implementation. |
I'm not against adding useful C tools to itertools, but you may have to
I'll take a look, but the question is how complex a C implementation |
C implementation speeds up the benchmark only 1.5x. Granting the fact that this idiom is used in stdlib less than two dozens times (without tests and iobench), I do not think more this is a worthful idea.
Definitely it will be more complex than for iter_unpack. ;) |
This seems reasonable to me to. So +1. Small bikeshed on the name: I think 'unpack_iter' would be more |
I thought so, but "unpack_iter" would mean we are unpacking an iterator, |
iter_unpack() is closer to how we name other iter variants, so I think you ought to stick with that. The other names are too creative (and hence less guessable). |
I think 'iter_unpack' is deceptive and wrong for the following reason. Up to now, 'ixyz' or 'iterxyz' or 'iter_xyz' has meant a version of 'xyz' that presents items one at a time rather than all at once in a collection object (usually in a list). Unpack returns a tuple, but the new function would *not* present the members of the tuple one at time. Instead, it would apply unpack multiple times, yielding multiple tuples. I would call the new thing 'unpack_all' or 'unpacker' (the latter works especially well for an iterator class). An unpacker is a person or machine that repeatedly unpacks. (I was once a bottle unpacker for a college summer job ;-). struct.unpacker(fmt, buffer)
Return an iterator that repeatedly unpacks successive buffer slices of size calcsize(fmt) into tuples according to the format string fmt. The buffer length must be an exact multiple of calcsize(fmt)). (? Not clear from text description. Add param to allow remainder?) |
New changeset d232cff25bbd by Antoine Pitrou in branch 'default': |
Ok, I don't want the bikeshedding to last too long, so I committed the patch with docs. Thanks everyone! |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: