This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: asynchat.async_chat and asyncore.dispatcher_with_send are not thread-safe
Type: behavior Stage:
Components: asyncio, Library (Lib) Versions: Python 3.6, Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: gvanrossum, ngg, vstinner, yselivanov
Priority: normal Keywords:

Created on 2016-02-16 20:47 by ngg, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (7)
msg260367 - (view) Author: NGG (ngg) Date: 2016-02-16 20:47
The initiate_send() method in both asynchat.async_chat and asyncore.dispatcher_with_send is not a thread-safe function, but it can be called from multiple threads.

For example if asyncore.loop() runs in a background thread, and the main thread sends a message then both threads will call this.

It can result in sending (part of) the message multiple times or not sending part of it.

Possible solution:
asyncore.dispatcher_with_send.out_buffer and asynchat.async_chat.producer_fifo should be guarded with a lock (it should be locked even in writable())
msg260368 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-02-16 20:53
asynchat is now deprecated in favor of asyncio.

Almost all asyncio functions are not thread-safe, it's now well documented, see the general info:
https://docs.python.org/dev/library/asyncio-dev.html#concurrency-and-multithreading

If we do something, I suggest to only touch asynchat doc to explain well that asyncore and asynchat are not thread-safe.
msg260369 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-02-16 20:58
I don't believe the original issue is a bug. There is nothing in the docs for asyncore or asynchat that suggests they would be thread-safe.
msg260380 - (view) Author: NGG (ngg) Date: 2016-02-17 08:25
If I want to write a TCP client which communicates back and forth with the server (both parties can send messages anytime) then it would be really easy to use it the following way:
Start a background thread with asyncore.loop(), and you can send messages easily, and handle_read() will be called automatically whenever data is received.
If this usage is not supported and I understand correctly then I can only send messages before starting the loop or callbacks from the loop (handle_accept, handle_read, etc).
This seems to be a much more difficult and error-prone way (at least for pre-Future python versions)
msg260381 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-02-17 09:07
"If this usage is not supported (...)"

You can use threads, but access concurrently asyncore/asynchat from different threads. You have to build a communicate channel between your threads using thread-safe primitive like queue.Queue. asyncio has a builtin support for that: loop.call_soon_threadsafe().

Why not using asyncio instead of having to rebuild your own implementation?
msg260383 - (view) Author: NGG (ngg) Date: 2016-02-17 09:16
"Why not using asyncio instead of having to rebuild your own implementation?"

The issue happened with python 2.7 and we don't want that project to depend on additional libraries.
I think we will upgrade that project to python3 soon and will probably use asyncio then.
msg260384 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-02-17 09:18
"I think we will upgrade that project to python3 soon and will probably use asyncio then."

Great :-) You may have a look at my https://pypi.python.org/pypi/sixer project, it may help :-)
History
Date User Action Args
2022-04-11 14:58:27adminsetgithub: 70559
2016-02-17 09:18:23vstinnersetmessages: + msg260384
2016-02-17 09:16:49nggsetmessages: + msg260383
2016-02-17 09:07:39vstinnersetmessages: + msg260381
2016-02-17 08:25:17nggsetmessages: + msg260380
2016-02-16 20:58:05gvanrossumsetstatus: open -> closed
resolution: not a bug
messages: + msg260369
2016-02-16 20:53:59vstinnersetnosy: + gvanrossum, vstinner, yselivanov
messages: + msg260368
components: + asyncio
2016-02-16 20:47:24nggcreate