Index: Doc/library/asynchat.rst =================================================================== --- Doc/library/asynchat.rst (revision 60780) +++ Doc/library/asynchat.rst (working copy) @@ -30,11 +30,12 @@ The :class:`asyncore.dispatcher` methods can be used, although not all make sense in a message/response context. - Like :class:`asyncore.dispatcher`, :class:`async_chat` defines a set of - events that are generated by an analysis of socket conditions after a - :cfunc:`select` call. Once the polling loop has been started the - :class:`async_chat` object's methods are called by the event-processing - framework with no action on the part of the programmer. + Like :class:`asyncore.dispatcher`, :class:`async_chat` defines a + set of events that are generated by an analysis of socket + conditions after a :cfunc:`select` call. Once the polling loop has + been started the :class:`async_chat` object's methods are called by + the event-processing framework with no action on the part of the + programmer. Two class attributes can be modified, to improve performance, or possibly even to conserve memory. @@ -49,19 +50,30 @@ The asynchronous output buffer size (default ``4096``). - Unlike :class:`asyncore.dispatcher`, :class:`async_chat` allows you to - define a first-in-first-out queue (fifo) of *producers*. A producer need - have only one method, :meth:`more`, which should return data to be - transmitted on the channel. - The producer indicates exhaustion (*i.e.* that it contains no more data) by - having its :meth:`more` method return the empty string. At this point the - :class:`async_chat` object removes the producer from the fifo and starts - using the next producer, if any. When the producer fifo is empty the - :meth:`handle_write` method does nothing. You use the channel object's - :meth:`set_terminator` method to describe how to recognize the end of, or - an important breakpoint in, an incoming transmission from the remote - endpoint. + Unlike :class:`asyncore.dispatcher`, :class:`async_chat` allows you + to define a first-in-first-out queue (fifo) of data sources. Data + sources come in three types: First, they can by simple bytes or + string values, added to the fifo by the :meth:`push` and + :meth:`push_str` methods, respectively. Second, they can be + iterables such that [bytes(x) for x in iter(*iterable*)] is valid, + added to the fifo by :meth:`push_iterable`, Finally, they can be + *producers*, added to the fifo with :meth:`push_with_producer`. + Producers are supported for backwards compatibility, and are + internally adapted into iterators. A producer need have only one + method, :meth:`more`, which should return data to be transmitted on + the channel, or b'' when it is exhausted. + + When a data source is exhausted, the :class:`async_chat` object + removes it from the fifo and starts using the next source, if + any. When the fifo is empty the :meth:`handle_write` method does + nothing. + + + You use the channel object's :meth:`set_terminator` method to + describe how to recognize the end of, or an important breakpoint + in, an incoming transmission from the remote endpoint. + To build a functioning :class:`async_chat` subclass your input methods :meth:`collect_incoming_data` and :meth:`found_terminator` must handle the data that the channel receives asynchronously. The methods are described @@ -70,17 +82,25 @@ .. method:: async_chat.close_when_done() - Pushes a ``None`` on to the producer fifo. When this producer is popped off + Pushes a ``None`` on to the source fifo. When this is popped off the fifo it causes the channel to be closed. .. method:: async_chat.collect_incoming_data(data) - Called with *data* holding an arbitrary amount of received data. The - default method, which must be overridden, raises a - :exc:`NotImplementedError` exception. + Called with *data* holding an arbitrary amount of received data in + the form of a bytes object. The default method stores the data in + an internal buffer, accesible by :meth:`get_data`. +.. method:: async_chat.get_data() + + Retrieves the data buffered by the default + :meth:`collect_incoming_data` since the last call to get_data, or + since the creation of the async_chat object if get_data has not + previously been called. + + .. method:: async_chat.discard_buffers() In emergencies this method will discard any data held in the input and/or @@ -89,10 +109,10 @@ .. method:: async_chat.found_terminator() - Called when the incoming data stream matches the termination condition set - by :meth:`set_terminator`. The default method, which must be overridden, - raises a :exc:`NotImplementedError` exception. The buffered input data - should be available via an instance attribute. + Called when the incoming data stream matches the termination + condition set by :meth:`set_terminator` or + :meth:`set_terminator_str`. The default method, which must be + overridden, raises a :exc:`NotImplementedError` exception. .. method:: async_chat.get_terminator() @@ -119,26 +139,46 @@ .. method:: async_chat.handle_write() - Called when the application may write data to the channel. The default - method calls the :meth:`initiate_send` method, which in turn will call - :meth:`refill_buffer` to collect data from the producer fifo associated - with the channel. + Called when the application may write data to the channel. The + default method calls the :meth:`initiate_send` method, which in + turn retrieves data from the source fifo and transmsits it through + the socket. .. method:: async_chat.push(data) - Creates a :class:`simple_producer` object (*see below*) containing the data - and pushes it on to the channel's ``producer_fifo`` to ensure its - transmission. This is all you need to do to have the channel write the - data out to the network, although it is possible to use your own producers - in more complex schemes to implement encryption and chunking, for example. + Pushes the passed data on to the channel's source fifo to ensure + its transmission, where *data* can be interpreted as a bytes + object. This is all you need to do to have the channel write the + data out to the network. +.. method:: async_chat.push_str(data, encoding) + + Pushes the passed data on to the channel's source fifo to ensure + its transmission, where *data* can be encoded into a bytes object + via the specified encoding. This is all you need to do to have the + channel write the data out to the network. + + It is wise to be aware the of endian behavior of your chosen + encoding, especially when communicating between computers of + different architectures. + + +.. method:: async_chat.push_iterable(iterable) + + Pushes the passed iterable onto the channel's source fifo to ensure + its transmission, where *iterable* results in a series of bytes + objects. This is all you need to do to have the channel write the + data out to the network. + + .. method:: async_chat.push_with_producer(producer) - Takes a producer object and adds it to the producer fifo associated with - the channel. When all currently-pushed producers have been exhausted the - channel will consume this producer's data by calling its :meth:`more` + Takes a producer object, adapts it into an iterator, and adds it to + the source fifo associated with the channel. When all + currently-pushed sources have been exhausted the channel will + consume this producer's data by repeatedly calling its :meth:`more` method and send the data to the remote endpoint. @@ -148,14 +188,6 @@ channels tested by the :cfunc:`select` loop for readability. -.. method:: async_chat.refill_buffer() - - Refills the output buffer by calling the :meth:`more` method of the - producer at the head of the fifo. If it is exhausted then the producer is - popped off the fifo and the next producer is activated. If the current - producer is, or becomes, ``None`` then the channel is closed. - - .. method:: async_chat.set_terminator(term) Sets the terminating condition to be recognized on the channel. ``term`` @@ -165,11 +197,11 @@ +-----------+---------------------------------------------+ | term | Description | +===========+=============================================+ - | *string* | Will call :meth:`found_terminator` when the | - | | string is found in the input stream | + | *bytes* | Will call :meth:`found_terminator` when the | + | | byte sequence is found in the input stream | +-----------+---------------------------------------------+ | *integer* | Will call :meth:`found_terminator` when the | - | | indicated number of characters have been | + | | indicated number of bytes have been | | | received | +-----------+---------------------------------------------+ | ``None`` | The channel continues to collect data | @@ -180,6 +212,18 @@ by the channel after :meth:`found_terminator` is called. +.. method:: async_chat.set_terminator_str(term, encoding) + + Sets the terminating condition to be recognized on the channel. + ``term`` is encoded into a bytes object using the ``encoding``, + after which it is treated just as a bytes object passed to + :meth:`set_terminator`. + + It is wise to be aware the of endian behavior of your chosen + encoding, especially when communicating between computers of + different architectures. + + .. method:: async_chat.writable() Should return ``True`` as long as items remain on the producer fifo, or the @@ -203,36 +247,6 @@ empty string. -.. class:: fifo([list=None]) - - Each channel maintains a :class:`fifo` holding data which has been pushed - by the application but not yet popped for writing to the channel. A - :class:`fifo` is a list used to hold data and/or producers until they are - required. If the *list* argument is provided then it should contain - producers or data items to be written to the channel. - - -.. method:: fifo.is_empty() - - Returns ``True`` if and only if the fifo is empty. - - -.. method:: fifo.first() - - Returns the least-recently :meth:`push`\ ed item from the fifo. - - -.. method:: fifo.push(data) - - Adds the given data (which may be a string or a producer object) to the - producer fifo. - - -.. method:: fifo.pop() - - If the fifo is not empty, returns ``True, first()``, deleting the popped - item. Returns ``False, None`` for an empty fifo. - The :mod:`asynchat` module also defines one utility function, which may be of use in network and textual analysis operations. @@ -270,25 +284,18 @@ asynchat.async_chat.__init__(self, conn=conn) self.addr = addr self.sessions = sessions - self.ibuffer = [] - self.obuffer = "" - self.set_terminator("\r\n\r\n") + self.set_terminator(b"\r\n\r\n") self.reading_headers = True self.handling = False self.cgi_data = None self.log = log - def collect_incoming_data(self, data): - """Buffer the data""" - self.ibuffer.append(data) - def found_terminator(self): if self.reading_headers: self.reading_headers = False - self.parse_headers("".join(self.ibuffer)) - self.ibuffer = [] - if self.op.upper() == "POST": - clen = self.headers.getheader("content-length") + self.parse_headers(self.get_data()) + if self.op.upper() == b"POST": + clen = self.headers.getheader(b"content-length") self.set_terminator(int(clen)) else: self.handling = True @@ -296,7 +303,6 @@ self.handle_request() elif not self.handling: self.set_terminator(None) # browsers sometimes over-send - self.cgi_data = parse(self.headers, "".join(self.ibuffer)) + self.cgi_data = parse(self.headers, self.get_data()) self.handling = True - self.ibuffer = [] self.handle_request() Index: Doc/library/asyncore.rst =================================================================== --- Doc/library/asyncore.rst (revision 60780) +++ Doc/library/asyncore.rst (working copy) @@ -118,8 +118,10 @@ .. method:: dispatcher.handle_expt() - Called when there is out of band (OOB) data for a socket connection. This - will almost never happen, as OOB is tenuously supported and rarely used. + Called when there is out of band (OOB) data or other exceptional + conditions for a socket connection. This will almost never happen, + as OOB is tenuously supported and rarely used. The default behavior + is to raise a socket.error describing the condition. .. method:: dispatcher.handle_connect() @@ -142,9 +144,9 @@ .. method:: dispatcher.handle_accept() - Called on listening channels (passive openers) when a connection can be - established with a new remote endpoint that has issued a :meth:`connect` - call for the local endpoint. + Called on listening channels (passive openers) when a connection + can be established with a new remote endpoint that has issued a + :meth:`connect` call for the local endpoint. .. method:: dispatcher.readable() Index: Lib/asynchat.py =================================================================== --- Lib/asynchat.py (revision 60780) +++ Lib/asynchat.py (working copy) @@ -32,18 +32,17 @@ the common internet protocols - smtp, nntp, ftp, etc..). The handle_read() method looks at the input stream for the current -'terminator' (usually '\r\n' for single-line responses, '\r\n.\r\n' -for multi-line output), calling self.found_terminator() on its -receipt. +'terminator', calling self.found_terminator() on its receipt. for example: -Say you build an async nntp client using this class. At the start -of the connection, you'll have self.terminator set to '\r\n', in -order to process the single-line greeting. Just before issuing a -'LIST' command you'll set it to '\r\n.\r\n'. The output of the LIST -command will be accumulated (using your own 'collect_incoming_data' -method) up to the terminator, and then control will be returned to -you - by calling your self.found_terminator() method. +Say you build an async nntp client using this class. At the start of +the connection, you'll have called self.set_terminator(b'\r\n'), in +order to process the single-line greeting. Just before issuing a LIST +command you'll set it to b'\r\n.\r\n'. The output of the LIST command +will be accumulated (using your own 'collect_incoming_data' method) up +to the terminator, and then control will be returned to you - by +calling your self.found_terminator() method. + """ import sys @@ -52,31 +51,155 @@ from collections import deque class async_chat (asyncore.dispatcher): - """This is an abstract class. You must derive from this class, and add - the two methods collect_incoming_data() and found_terminator()""" + """This is an abstract class. You must derive from this class, + and add the `found_terminator` method. Children may also override + `collect_incoming_data` to override the default buffering of + incoming data. + + The set_terminator(term) or set_terminator_str(term, encoding) + method almost always should be called before entering + asyncore.loop. + + In addition to overriding methods, children may override the + variables `ac_in_buffer_size` and `ac_out_buffer_size`, which + provide hints regarding the number of bytes that may be stored in + the incoming and outgoing buffers, respectively. Changing these + will not affect functionality, but may alter performance. The + default value of each is 4096. + + """ + # these are overridable defaults ac_in_buffer_size = 4096 ac_out_buffer_size = 4096 def __init__ (self, conn=None): - self.ac_in_buffer = b'' - self.ac_out_buffer = b'' - self.producer_fifo = fifo() + """Constructor, should be called by child's __init__ + + Optional arguments: + conn -- An already initialized socket. + + conn, if provided and not None, will be used as the socket for + this channel. If conn is provided, create_socket and its + related methods need not be called. + + """ + + # for terminator matching + self.__in_buffer = b'' + # we use a list here rather than cStringIO for a few reasons... + # del lst[:] is faster than sio.truncate(0) + # lst = [] is faster than sio.truncate(0) + # cStringIO will be gaining unicode support in py3k, which + # will negatively affect the performance of bytes compared to + # a ''.join() equivalent + self.__incoming = [] + + # we toss the use of the "simple producer" and replace it with + # a pure deque, which the original fifo was a wrapping of + self.__producer_fifo = deque() + asyncore.dispatcher.__init__ (self, conn) def collect_incoming_data(self, data): - raise NotImplementedError("must be implemented in subclass") + """Store received data for later processing + Arguments: + data -- A bytes object representing some received data. + + collect_incoming_data may be overridden in a child class. The + default implementation simply records incoming data, making it + accessible via the `get_data` method; `found_terminator` is + the proper place to do processing in most cases. + + """ + self.__incoming.append(data) + + def get_data(self): + """Retrieve the data received so far. Returns a bytes object. + + This method is tied to the default collect_incoming_data + implementation. It retrieves and empties the buffer that is + filled by the default collect_incoming_data. + + """ + d = b''.join(self.__incoming) + del self.__incoming[:] + return d + def found_terminator(self): - raise NotImplementedError("must be implemented in subclass") + """Process a complete section of data received from the socket + found_terminator must be overridden in a child class for + async_chat to be useful. It is called when the terminator is + found in the received data stream, after calling + `collect_incoming_data` with all bytes prior to the terminator + in the stream. The terminator is defined by the most recent + call to `set_terminator` or `set_terminator_str`. If + `collect_incoming_data` has not been overridden, get_data() + will return the data received up to the terminator. + + + """ + raise NotImplementedError("found_terminator must be implemented in subclass") + def set_terminator (self, term): - "Set the input delimiter. Can be a fixed string of any length, an integer, or None" - self.terminator = term + """Set the input delimiter. + Arguments: + term -- Can be a bytes object, an integer, or None. + + If term is None, async_chat will never call + found_terminator. This is occasionally useful, if + `collect_incoming_data` performs all necessary handling of the + data. + + If term is an int object, async_chat will read that many bytes + from the data stream, and then call `found_terminator`. This + is useful for reading protocol sections of known length. + + If term is neither an int nor None, it will be passed to the + bytes object constructor. async_chat will look for that + sequence of bytes in the incoming data stream, and call + `found_terminator` when it is discovered. This is useful for + reading protocol sections of varying or unknown length. + + set_terminator may be called as many times as you like. This + is quite often useful -- for example in HTTP you might read + the headers by doing set_terminator(b'\n\n') and then extract + the content length from the headers and do + set_terminator(content_length) to read the page data. + + """ + + if term is None: + self.terminator = None + elif isinstance(term, int): + self.terminator = term + else: + self.terminator = bytes(term) + + def set_terminator_str(self, term, encoding): + """Set the input delimiter to an encoded string + + Arguments: + term -- A str object. + encoding -- The name of a codec. + + This method is like `set_terminator`, except that it sets the + terminator to a sequence of bytes representing term in the + given encoding. + + It is very important to be aware of byte ordering when using + this function. Some encodings produce endian-sentive results. + + """ + self.set_terminator(term.encode(encoding)) + def get_terminator (self): + """Return the current terminator""" return self.terminator # grab some more data from the socket, @@ -85,39 +208,39 @@ # if found, transition to the next state. def handle_read (self): + """asyncore function. Overriding children must invoke.""" try: - data = self.recv (self.ac_in_buffer_size) - except socket.error as why: + data = self.recv(self.ac_in_buffer_size) + except socket.error: self.handle_error() return - if isinstance(data, str): - data = data.encode('ascii') - self.ac_in_buffer = self.ac_in_buffer + bytes(data) + data = bytes(data) + self.__in_buffer = self.__in_buffer + data - # Continue to search for self.terminator in self.ac_in_buffer, + # Continue to search for self.terminator in self.__in_buffer, # while calling self.collect_incoming_data. The while loop # is necessary because we might read several data+terminator - # combos with a single recv(1024). + # combos with a single recv(4096). - while self.ac_in_buffer: - lb = len(self.ac_in_buffer) + while self.__in_buffer: + lb = len(self.__in_buffer) terminator = self.get_terminator() if not terminator: # no terminator, collect it all - self.collect_incoming_data (self.ac_in_buffer) - self.ac_in_buffer = b'' - elif isinstance(terminator, int) or isinstance(terminator, int): + self.collect_incoming_data (self.__in_buffer) + self.__in_buffer = b'' + elif isinstance(terminator, int): # numeric terminator n = terminator if lb < n: - self.collect_incoming_data (self.ac_in_buffer) - self.ac_in_buffer = b'' + self.collect_incoming_data (self.__in_buffer) + self.__in_buffer = b'' self.terminator = self.terminator - lb else: - self.collect_incoming_data (self.ac_in_buffer[:n]) - self.ac_in_buffer = self.ac_in_buffer[n:] + self.collect_incoming_data (self.__in_buffer[:n]) + self.__in_buffer = self.__in_buffer[n:] self.terminator = 0 self.found_terminator() else: @@ -129,122 +252,206 @@ # 3) end of buffer does not match any prefix: # collect data terminator_len = len(terminator) - if isinstance(terminator, str): - terminator = terminator.encode('ascii') - index = self.ac_in_buffer.find(terminator) + index = self.__in_buffer.find(terminator) if index != -1: # we found the terminator if index > 0: # don't bother reporting the empty string (source of subtle bugs) - self.collect_incoming_data (self.ac_in_buffer[:index]) - self.ac_in_buffer = self.ac_in_buffer[index+terminator_len:] + self.collect_incoming_data (self.__in_buffer[:index]) + self.__in_buffer = self.__in_buffer[index+terminator_len:] # This does the Right Thing if the terminator is changed here. self.found_terminator() else: # check for a prefix of the terminator - index = find_prefix_at_end (self.ac_in_buffer, terminator) + index = find_prefix_at_end (self.__in_buffer, terminator) if index: if index != lb: # we found a prefix, collect up to the prefix - self.collect_incoming_data (self.ac_in_buffer[:-index]) - self.ac_in_buffer = self.ac_in_buffer[-index:] + self.collect_incoming_data (self.__in_buffer[:-index]) + self.__in_buffer = self.__in_buffer[-index:] break else: # no prefix, collect it all - self.collect_incoming_data (self.ac_in_buffer) - self.ac_in_buffer = b'' + self.collect_incoming_data (self.__in_buffer) + self.__in_buffer = b'' def handle_write (self): - self.initiate_send () + """asyncore function. Overriding children must invoke.""" + self.initiate_send() def handle_close (self): + """asyncore function. Overriding children must invoke.""" self.close() def push (self, data): - self.producer_fifo.push (simple_producer (data)) + """Place data in the outgoing queue. + + Arguments: + data -- The expression bytes(data) must be meaningful + + Appends bytes(data) to the sequence of bytes to be sent + through the socket. Each time asyncore.loop discovers that + the socket is writable, some or all of the bytes in that queue + will be transmitted. + + """ + sabs = self.ac_out_buffer_size + data = bytes(data) + if len(data) > sabs: + for i in xrange(0, len(data), sabs): + self.__producer_fifo.append(data[i:i+sabs]) + else: + self.__producer_fifo.append(data) self.initiate_send() - def push_with_producer (self, producer): - self.producer_fifo.push (producer) + def push_str(self, data, encoding): + """Place a string in the outgoing queue. + + Arguments: + data -- The expression data.encode(encoding) must be meaningful + + Appends bytes(data.encode(encoding)) to the sequence of bytes + to be sent through the socket. Each time asyncore.loop + discovers that the socket is writable, some or all of the + bytes in that queue will be transmitted. + + It is very important to be aware of byte ordering when using + this function. Some encodings produce endian-sentive results. + + """ + self.push(data.encode(encoding)) + + def push_iterable(self, iterable): + """Place the iterated items in the outgoing queue + + Arguments: + iterable -- [bytes(x) for x in iterable] must be meaningful + + Appends bytes(x) for each x in iterator to the sequence of + bytes to be sent through the socket. Each time asyncore.loop + discovers that the socket is writable, some or all of the + bytes in that queue will be transmitted. + + """ + self.__producer_fifo.append(iter(iterable)) self.initiate_send() + def push_with_producer (self, producer): + """Place a producer in the outgoing queue. + + Arguments: + producer -- The expression producer.more() must be meaningful + + Repeatedly appends bytes(producer.more()) to the sequence of + bytes to be sent through the socket, until + bool(producer.more()) is False. Each time asyncore.loop + discovers that the socket is writable, some or all of the + bytes in that queue will be transmitted. + + """ + def adapter(prod): + while True: + data = prod.more() + if data == b'': + raise StopIteration + yield data + + self.push_iterable(adapter(producer)) + def readable (self): "predicate for inclusion in the readable for select()" - return (len(self.ac_in_buffer) <= self.ac_in_buffer_size) + # cannot use the old predicate, it violates the claim of the + # set_terminator method. + # return (len(self.__in_buffer) <= self.ac_in_buffer_size) + return 1 + def writable (self): "predicate for inclusion in the writable for select()" - # return len(self.ac_out_buffer) or len(self.producer_fifo) or (not self.connected) - # this is about twice as fast, though not as clear. - return not ( - (self.ac_out_buffer == b'') and - self.producer_fifo.is_empty() and - self.connected - ) + return self.__producer_fifo or (not self.connected) def close_when_done (self): "automatically close this channel once the outgoing queue is empty" - self.producer_fifo.push (None) + self.__producer_fifo.append(None) - # refill the outgoing buffer by calling the more() method - # of the first producer in the queue - def refill_buffer (self): - while 1: - if len(self.producer_fifo): - p = self.producer_fifo.first() - # a 'None' in the producer fifo is a sentinel, - # telling us to close the channel. - if p is None: - if not self.ac_out_buffer: - self.producer_fifo.pop() - self.close() - return - elif isinstance(p, str) or isinstance(p, bytes): - if isinstance(p, str): - p = p.encode('ascii') - self.producer_fifo.pop() - self.ac_out_buffer = self.ac_out_buffer + p - return - data = p.more() - if data: - if isinstance(data, str): - data = data.encode('ascii') - self.ac_out_buffer = self.ac_out_buffer + bytes(data) - return - else: - self.producer_fifo.pop() - else: + def initiate_send (self): + """Internal function. Do not invoke or override.""" + while self.__producer_fifo and self.connected: + source = self.__producer_fifo.popleft() + + # Handle None entry. + if source is None: + self.handle_close() return - def initiate_send (self): - obs = self.ac_out_buffer_size - # try to refill the buffer - if (len (self.ac_out_buffer) < obs): - self.refill_buffer() + # Handle empty entries. + if not source: + continue - if self.ac_out_buffer and self.connected: - # try to send the buffer try: - num_sent = self.send (self.ac_out_buffer[:obs]) - if num_sent: - self.ac_out_buffer = self.ac_out_buffer[num_sent:] + data = source.__next__() + self.__producer_fifo.appendleft(source) + except StopIteration: + continue + except AttributeError: + data = source - except socket.error as why: + data = bytes(data) + + # Send the data. + try: + num_sent = self.send(data[:self.ac_out_buffer_size]) + except socket.error: self.handle_error() return + if num_sent: + if num_sent < len(data): + self.__producer_fifo.appendleft(data[num_sent:]) + + # We tried to send some actual data. + return + def discard_buffers (self): + """Discard all incoming and outgoing data. + + It is generally wise to avoid the use of this function, as it + results in data loss. This is rarely useful. + + """ # Emergencies only! - self.ac_in_buffer = b'' - self.ac_out_buffer = b'' - while self.producer_fifo: - self.producer_fifo.pop() + self.__in_buffer = b'' + del self.__incoming[:] + self.__producer_fifo.clear() - class simple_producer: + """Produce bytes for transmission. + + This is a very basic producer; the only extra functionality it + provides over async_chat.push(bytes) is to break the data up into + blocks of arbitrary size, such that each call to more produces a + block of data no larger than the requested size. Even that is not + very useful, since setting async_chat.ac_out_buffer_size achieves + the same thing. + + """ + def __init__ (self, data, buffer_size=512): - self.data = data + """Initialize the producer with a byte sequence + + Arguments: + data -- bytes(data) must be meaningful. + + Optional arguments: + buffer_size -- A number, defaults to 512 + + The simple_producer will pop and return blocks of `data` of + length min(`buffer_size`, len(`data`)) until data is empty. + + """ + + self.data = bytes(data) self.buffer_size = buffer_size def more (self): @@ -257,31 +464,6 @@ self.data = b'' return result -class fifo: - def __init__ (self, list=None): - if not list: - self.list = deque() - else: - self.list = deque(list) - - def __len__ (self): - return len(self.list) - - def is_empty (self): - return not self.list - - def first (self): - return self.list[0] - - def push (self, data): - self.list.append(data) - - def pop (self): - if self.list: - return (1, self.list.popleft()) - else: - return (0, None) - # Given 'haystack', see if any prefix of 'needle' is at its end. This # assumes an exact match has already been checked. Return the number of # characters matched. @@ -298,6 +480,17 @@ # regex: 14035/s def find_prefix_at_end (haystack, needle): + """Returns > 0 if the beginning of needle is at the end of haystack + + Arguments: + haystack -- A bytes object. + needle -- A bytes object. + + Given haystack, see if any prefix of needle is at its end. This + assumes an exact match has already been checked. Return the + number of characters matched. + + """ l = len(needle) - 1 while l and not haystack.endswith(needle[:l]): l -= 1 Index: Lib/smtpd.py =================================================================== --- Lib/smtpd.py (revision 60780) +++ Lib/smtpd.py (working copy) @@ -123,12 +123,12 @@ self.__fqdn = socket.getfqdn() self.__peer = conn.getpeername() print('Peer:', repr(self.__peer), file=DEBUGSTREAM) - self.push('220 %s %s' % (self.__fqdn, __version__)) - self.set_terminator('\r\n') + self.push_str('220 %s %s' % (self.__fqdn, __version__), 'ascii') + self.set_terminator(b'\r\n') # Overrides base class for convenience def push(self, msg): - asynchat.async_chat.push(self, msg + '\r\n') + asynchat.async_chat.push(self, msg + b'\r\n') # Implementation of base class abstract method def collect_incoming_data(self, data): @@ -141,7 +141,7 @@ self.__line = [] if self.__state == self.COMMAND: if not line: - self.push('500 Error: bad syntax') + self.push_str('500 Error: bad syntax', 'ascii') return method = None i = line.find(' ') @@ -153,13 +153,13 @@ arg = line[i+1:].strip() method = getattr(self, 'smtp_' + command, None) if not method: - self.push('502 Error: command "%s" not implemented' % command) + self.push_str('502 Error: command "%s" not implemented' % command, 'ascii') return method(arg) return else: if self.__state != self.DATA: - self.push('451 Internal confusion') + self.push_str('451 Internal confusion', 'ascii') return # Remove extraneous carriage returns and de-transparency according # to RFC 821, Section 4.5.2. @@ -177,32 +177,32 @@ self.__rcpttos = [] self.__mailfrom = None self.__state = self.COMMAND - self.set_terminator('\r\n') + self.set_terminator(b'\r\n') if not status: - self.push('250 Ok') + self.push_str('250 Ok', 'ascii') else: - self.push(status) + self.push_str(status, 'ascii') # SMTP and ESMTP commands def smtp_HELO(self, arg): if not arg: - self.push('501 Syntax: HELO hostname') + self.push_str('501 Syntax: HELO hostname', 'ascii') return if self.__greeting: - self.push('503 Duplicate HELO/EHLO') + self.push_str('503 Duplicate HELO/EHLO', 'ascii') else: self.__greeting = arg - self.push('250 %s' % self.__fqdn) + self.push_str('250 %s' % self.__fqdn, 'ascii') def smtp_NOOP(self, arg): if arg: - self.push('501 Syntax: NOOP') + self.push_str('501 Syntax: NOOP', 'ascii') else: - self.push('250 Ok') + self.push_str('250 Ok', 'ascii') def smtp_QUIT(self, arg): # args is ignored - self.push('221 Bye') + self.push_str('221 Bye', 'ascii') self.close_when_done() # factored @@ -223,49 +223,49 @@ print('===> MAIL', arg, file=DEBUGSTREAM) address = self.__getaddr('FROM:', arg) if arg else None if not address: - self.push('501 Syntax: MAIL FROM:
') + self.push_str('501 Syntax: MAIL FROM:', 'ascii') return if self.__mailfrom: - self.push('503 Error: nested MAIL command') + self.push_str('503 Error: nested MAIL command', 'ascii') return self.__mailfrom = address print('sender:', self.__mailfrom, file=DEBUGSTREAM) - self.push('250 Ok') + self.push_str('250 Ok', 'ascii') def smtp_RCPT(self, arg): print('===> RCPT', arg, file=DEBUGSTREAM) if not self.__mailfrom: - self.push('503 Error: need MAIL command') + self.push_str('503 Error: need MAIL command', 'ascii') return address = self.__getaddr('TO:', arg) if arg else None if not address: - self.push('501 Syntax: RCPT TO: ') + self.push_str('501 Syntax: RCPT TO: ', 'ascii') return self.__rcpttos.append(address) print('recips:', self.__rcpttos, file=DEBUGSTREAM) - self.push('250 Ok') + self.push_str('250 Ok', 'ascii') def smtp_RSET(self, arg): if arg: - self.push('501 Syntax: RSET') + self.push_str('501 Syntax: RSET', 'ascii') return # Resets the sender, recipients, and data, but not the greeting self.__mailfrom = None self.__rcpttos = [] self.__data = '' self.__state = self.COMMAND - self.push('250 Ok') + self.push_str('250 Ok', 'ascii') def smtp_DATA(self, arg): if not self.__rcpttos: - self.push('503 Error: need RCPT command') + self.push_str('503 Error: need RCPT command', 'ascii') return if arg: - self.push('501 Syntax: DATA') + self.push_str('501 Syntax: DATA', 'ascii') return self.__state = self.DATA - self.set_terminator('\r\n.\r\n') - self.push('354 End data with