There was some discussion on python-ideas about this, and I figured it would be more productive to bring it here since to me this appears to be a glaring omission.

The struct module has no capability to support variable-length strings; this includes null-terminated and Pascal-ish strings with a different integer datatype (usually in binary) specifying length.

This unfortunate omission makes the struct module extremely unwieldy to use in situations where you need to unpack a lot of variable-length strings, especially iteratively; see https://mail.python.org/pipermail/python-ideas/2017-January/044328.html for why. For zero-terminated strings, it is essentially impossible.

It's worth noting many modern protocols use variable-length strings, including DHCP.

I therefore propose the following extensions to the struct module (details can be bikeshedded over :P):

- Z (uppercase) format specifier (I did not invent this idea, see https://github.com/stendec/netstruct - although that uses $), which states the preceding whole-number datatype is the length of a string that follows.
- z (lowercase) format specifier, which specifies a null-terminated (also known as C style) string. An optional length parameter can be added to specify the maximum search length.

These two additions will make the struct module much more usable in a wider variety of contexts.

Could you provide some examples of using these format specifiers? I suppose that due to limitations of the struct module the way in which they can be implemented would be not particularly useful for you.

IMO, as one of the previous maintainers of the struct module, this feature request isn't compatible with the current design and purpose of the struct module. I agree that there's an important problem to solve (and one I've had to solve many times for various formats in consulting work); it's simply that the struct module isn't the right place to solve it.

From Yury Selivanov:
-------------------
This is a neat idea, but this will only work for parsing framed
binary protocols.  For example, if you protocol prefixes all packets
with a length field, you can write an efficient read buffer and
use your proposal to decode all of message's fields in one shot.
Which is good.

Not all protocols use framing though.  For instance, your proposal
won't help to write Thrift or Postgres protocols parsers.

Overall, I'm not sure that this is worth the hassle.  With proposal:

   data, = struct.unpack('!H$', buf)
   buf = buf[2+len(data):]

with the current struct module:

   len, = struct.unpack('!H', buf)
   data = buf[2:2+len]
   buf = buf[2+len:]

Another thing: struct.calcsize won't work with structs that use
variable length fields.

Ethan, thanks for moving my reply on the list to here.  Also +1 to what Mark said.

FWIW, I concur with Mark and Yuri that the feature request isn't compatible with the design and purpose of the struct module

To add a bit to what Yury said, even framing isn't always compatible with this proposal.  For example, in dask/distributed, we first have a word for the number of frames, then one word per frame to indicate each frame's length, then the frame bodies.

Hi,

After discussing this on the python-ideas ML a bit more, this is actually a feature people want a great deal. It can't cover every use case, but to expand it further than this proposal and make it do so is way beyond the scope of this proposal.

It may not be completely useful for every protocol, but it is sufficiently useful for many people who have simpler use cases. There is no reason to prevent the addition of this feature other than what boils down to "well code will have to be written..."

I don't buy the argument that it's "outside the scope of the module" more than I think it's more "I don't like the idea of struct being used for non-fixed data." C structures support zero-terminated char arrays, and there is already a Pascal string type.

I didn't realise there'd be this much opposition to just adding two format specifiers... :/

Also, to add to the discussion:

* Rejecting this because "it doesn't cover every use case" is a red herring at best. If this can't cover your use case, odds are the struct module can *never* cover it. That is no reason to reject it alone; you would need something more heavyweight than the struct module anyway.

* If the module can't cover your use case with this feature, it can't cover it right now, so why obstruct it? It covers my use cases for it just fine.

* Not everyone needs something more heavyweight, or wants to import some bigger module just because they need variable-length strings.

* If the real goal is to discourage use of the struct module, too bad. People are actually using it in production and it serves its (rather small) purpose very well. Other people would like to use the module for their use cases, but presently cannot, and this proposal would help cover their particular cases.

* The fact that the netstruct module exists with this feature is proof enough there's demand; not to mention the discussion on the python-ideas ML shows that many people already would find this very useful. It's not like I'm proposing adding braces or some horrible huge proposal, I'm adding two format specifiers. *Two.*

A couple of questions that haven't been brought up yet:

1. Do you have any thoughts on how alignment should behave for '@'-style structs containing variable-length strings? I suspect the easiest solution may be simply to disallow that combination, and only allow variable-length strings for "standard" struct types (those with a format string starting with one of "=", "<", ">", "!"), where alignment isn't an issue.

2. For the Struct object, what should the .size attribute give for a variable-length struct? (Or should accessing the attribute raise an exception?)

3. Any thoughts about how the internal representation of the Struct object would need to change? I guess you'd want to drop the "offset" field of the "formatcode" struct, and compute the offsets on the fly during packing and unpacking (or would you try to keep the offset for the non-variable-length cases?). You'd probably also want to find a way to encode the information about whether the struct is variable-length or not in the PyStructObject struct. A key requirement here is that there should be no substantial performance regression for packing / unpacking with structs that don't use the variable-length feature. It doesn't seem likely to me that getting rid of the precalculated offset would cause such a regression, but it's something that should be checked.

If you want to add the support variable-size fields, this is incompatible with the struct module design.

If you want to add the support of variable-length strings inside fixed-size fields (as with the 'p' format unit), I think this case is not enough common.

After getting so much negative responses from core developers well-versed in internals of the struct module, I think this issue should be closed.

FWIW, the existence of netstruct https://github.com/stendec/netstruct does at least show that a coherent proposal is possible (the docs provide an API, motivation and examples).

One question is whether this could be implemented in the stdlib struct module without major brain surgery to the existing code which is organized around an initial prepare_s() step whose output is a PyStructObject which includes a fixed s_size field.  The various parsing routines all depend on the s_size field.  Where this proposal would require too much of a total rewrite is likely only answerable by someone building a patch with tests and real-world examples.

Another question is whether supporting variable length input would be better served by providing an alternative API that is better designed for it.  For example, unpack(fmt, input_stream or iterator, limit=None) which would consume as many bytes as needed from the input stream with an optional limit.  This would be nicer than having to figure-out in advance how to extract an input string of the appropriate size.

A last question is whether this should remain outside the standard library.  With PyPI becoming so rich and easy to access, we're often deciding that code is better-off outside the standard library where it can flourish in a more fluid environment.

s/Where/Whether/

I am closing this issue as "rejected", given the consensus that writing a patch could be a major undertaking, the lack of such a patch, and the fact that there has been no activity on the issue thread (or the python-ideas mailing list) for nearly 5 years.