xdrlib currently requires the entire operation
to be done at once; you
store all your objects, then get
back the string; then later, have the
entire string to present to the decoder
before anything is decoded.

This makes it impractical to handle arbitrarily
large files of xdr data, for instance. I imagine
XDR was intended for network packets and
so forth, but it would be nice to use as a binary
mode pickle-type thing which also happens
to be compatible with an RFC...

Anyway, what I am suggesting is that the xdrlib classes
could be rearranged to allow blocking/unblocking of the data stream
through subclassing. For a decoder, the unpack_ methods
would check if there was enough data, and if not, call
a member function (intended to be overloaded) which would
get more data. It would then be a relatively simple matter
to create a subclass which reads from a file.

To be more specific: instead of 

    def unpack_float(self):
        i = self.__pos
        self.__pos = j = i+4
        data = self.__buf[i:j]
        if len(data) < 4:
            raise EOFError
        return struct.unpack('>f', data)[0]

.... it would be

    def unpack_double(self):
        return struct.unpack('>d', self.getmore(8))[0]

    def unpack_float(self):
        return struct.unpack('>f', self.getmore(4))[0]

    def getmore(self,n):
        i = self.__pos
        self.__pos = j = i+n
        data = self.__buf[i:j]
        if len(data) < n:
            raise EOFError
        return data

... which is compatible with current behaviour; by overloading
getmore you can arrange to read efficiently from a file object
or whatever. I volunteer to do the recoding if you want...




  

Logged In: YES 
user_id=21627

This would only apply to the Unpacker, right? I'd propose to
call this method 'read'. Please note that everybody
implementing this method also needs to implement
get_/set_position(), get_buffer(), and done().

The general idea sounds reasonable to me. Would you like to
work on a patch implementing it (ideally with documentation
and test cases)?

Logged In: YES 
user_id=292741

Well, it could apply to the packer too. for a file writer, you would
do an arbitrary # of of pack operations; each time the
data packed exceeded a certain size, a nice-sized chunk
would be written out to the underlying file. I'll have to think
about the actual subclassing interface a bit more.

regarding get_/set_position(), get_buffer(), and done() - these need to be
implemented when the unpacker is subclassed, but the implementation need not
be useful. I.e. you want to prevent the normal versions of these from being called,
but if you don't intend to support them, the implementation you provide could be
just a 'raise'.

And yes, 'read' makes sense since it does the same thing as file.read().

Things got busy in the last little while - new baby! but I'll take a look again
when I get a chance. I think the mods to the unpacker are pretty simple. And
I'll get the existing test code and look at that too.

Seems like a very rare use case, closing is suggested.