classification
Title: asynchat forgets packets when push is called from a thread
Type: behavior
Components: Library (Lib) Versions: Python 2.6, Python 2.5
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: josiahcarlson Nosy List: giampaolo.rodola, josiah.carlson, josiahcarlson, xix xeaon
Priority: Keywords:

Created on 2008-05-10 11:00 by xix xeaon, last changed 2008-07-03 18:19 by josiahcarlson.

Files
File name Uploaded Description Edit Remove
asyn-delay.py xix xeaon, 2008-05-10 11:00 code showing the bug
Messages
msg66510 (view) Author: xix xeaon (xix xeaon) Date: 2008-05-10 10:59
okay, my first bug post, hope I do it right.

if you push more than 512 bytes (issue2073) from a thread then only the
first packet will be sent, all other packets will be forgotten.

the remaining packets will be sent when something else happens, like a
new connection is opened, or more input data is read. it's also sent
when the select times out (default 30 seconds) and starts over.

if push is called twice then the first two packets will be sent but the
rest will be forgotten.

if this problem arises in your code then you can easily fix it with a 0
second timeout for loop(), or preferably something like 0.0001 so that
the cpu stays calm ;)

you should be able to reproduce this bug with the attached file, just
run it and then connect with netcat like:
echo "\n" | nc localhost 1100
netcat will print almost 6 and a half lines of x and then the rest after
5 seconds (30 seconds is boring..) or if you have wireshark up then
you'll see it easily.

and would someone please apply all the patches for this module! =)
msg67167 (view) Author: Josiah Carlson (josiahcarlson) Date: 2008-05-21 19:00
My suggestion: don't do that.  Asynchronous sockets, and
asyncore/related libraries are not designed for, nor intended to be used
as part of a threaded IO application.  Why?  Because most protocols are
very concerned with data ordering, and sending from multiple threads can
cause *serious* issues.  I do not believe that this should change.

Note that you can work around this limitation by using something like
the following, but again, this is not suggested (you should instead work
asyncore.poll() calls into some sort of main loop within your application).

from Queue import Queue

check_threads = 0

class my_async(asyncore.dispatcher):
    def __init__(self, *args, **kwargs):
        self.q = Queue()
        asyncore.dispatcher.__init__(self, *args, **kwargs)
    def push_t(self, data):
        global check_threads
        self.q.put(data)
        check_threads = 1
    def handle_threaded_push(self):
        while not self.q.empty():
            self.push(self.q.get())

def loop(timeout=.1, map=None):
    global check_threads
    if map is None:
        map = asyncore.socket_map
    while 1:
        asyncore.poll(timeout, map)
        if check_threads:
            check_threads = 0
            for v in map.values():
                try:
                    v.handle_threaded_push()
                except:
                    #handle exceptions better here
                    pass
msg67175 (view) Author: xix xeaon (xix xeaon) Date: 2008-05-21 22:03
I have good reason to use this combination of asynchat and threads (in a
way which doesn't cause any of these serious issues you speak of), but
if you don't want it fixed then that's fine with me since I know how to
work around it, I just reported it =P
History
Date User Action Args
2008-07-03 18:19:40josiahcarlsonsetstatus: pending -> closed
2008-05-21 22:04:01xix xeaonsetmessages: + msg67175
2008-05-21 19:00:46josiahcarlsonsetstatus: open -> pending
assignee: josiahcarlson
resolution: wont fix
messages: + msg67167
2008-05-20 18:33:09giampaolo.rodolasetnosy: + josiahcarlson, giampaolo.rodola, josiah.carlson
2008-05-10 11:00:24xix xeaoncreate