classification
Title: Document thread safety
Type: Stage: resolved
Components: Documentation Versions: Python 3.8
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, rhettinger, sir-sigurd, vstinner
Priority: normal Keywords:

Created on 2018-11-19 11:50 by vstinner, last changed 2021-09-21 22:22 by vstinner. This issue is now closed.

Messages (8)
msg330093 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-19 11:50
Many developers only discover that a Python function/module is not thread safe because they have a bug in production...

Some examples:

* bpo-7672: ssl
* bpo-8865: select.poll is not thread safe
* bpo-539175, bpo-21216: socket.gethostbyname()
* bpo-7980: time.strptime()
* bpo-6647: warnings.catch_warnings()
* bpo-11077, bpo-33479: Tkinter
* bpo-1336, bpo-19809: subprocess on Python 2
* bpo-15329: deque
* bpo-35275: os.umask()

Hopefully, sometimes it was possible to fix it:

* bpo-3139: bytearray, buffer protocol
* bpo-28969: @functools.lru_cache
* bpo-21291: subprocess.Popen.wait()

In the asyncio documentation, I explicitly documented that, by design, most classes are not thread-safe. For example, asyncio.Lock() is *NOT* thread-safe:
https://docs.python.org/dev/library/asyncio-sync.html#asyncio.Lock

Maybe we should start to use a standard way to describe "thread safety". See "POSIX Safety Concepts" of the GNU libc:

https://www.gnu.org/software/libc/manual/html_node/POSIX-Safety-Concepts.html#POSIX-Safety-Concepts

Example with setlocale, "MT-Unsafe":

https://www.gnu.org/software/libc/manual/html_node/Setting-the-Locale.html

--

My own (incomplete) list of "process-wide states":
https://vstinner.readthedocs.io/threads.html#process-wide
msg330095 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-19 11:53
"Proces-wide" vs "thread-safe" is one thing. Another interesting property is "reentrant", but maybe that's too wide?

Somehow related, iterating on a container and modify it may or may not work depending on the container type and the kind of modifications.

--

> bpo-8865: select.poll is not thread safe

I checked select documentation, it doesn't mention "thread" anywhere :-(

https://docs.python.org/dev/library/select.html#poll-objects
msg330110 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2018-11-19 19:14
The general rule in Python is that nothing can be considered atomic unless specifically documented as atomic (such as the queue module or lru_cache which have internal locks).  The only safe action is to put locks around all accesses to shared resources.  We should have a FAQ entry to that effect.  It should also note that "thread-safe" means different things to different people.
msg330117 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2018-11-19 22:13
One idea is that you could author a threading HOWTO document that covers atomicity, locks, queues, reentrancy, etc.  This is a thorny topic and it would be nice to have the principles and techniques collected in one place.

Ideally, it would include examples of what to expect in various situations.  For example, the pure python OrderedDict can be put in an inconsistent state if two threads make updates without a mutex; however, the containers implemented in C can never be broken even if they don't guarantee atomicity (i.e. a dict update making a pure python callback to __hash__ will never result in a broken dict).

ISTM that the docs have presumed that people using threading know what they're doing; however, we know that isn't always true ;-)
msg330124 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-20 10:21
I don't think that it would make sense to annotate *each* function in the documentation, that would require too much work. I suggest to start to annotate functions which are known to have "side effects" or are not safe whereas users can be surprised to read that they are unsafe in the context of multithreading.

For example, document that os.chdir() is "process-wide" (I'm not sure how to document it). I think that everybody agrees that it's the case, and IMHO it's an useful information. At least, one user has been surprised to read that os.umask() is process-wide:

"I would be great, if the python standard library would provide correspondig thread safe method."
https://bugs.python.org/issue35275

> One idea is that you could author a threading HOWTO document that covers atomicity, locks, queues, reentrancy, etc.  This is a thorny topic and it would be nice to have the principles and techniques collected in one place.

Yes. About the definition of atomicity, thread-safe, process-wide, etc. There is an existing example: contextlib describes "Reentrant context managers".
https://docs.python.org/dev/library/contextlib.html#reentrant-cms

> Ideally, it would include examples of what to expect in various situations.  For example, the pure python OrderedDict can be put in an inconsistent state if two threads make updates without a mutex; however, the containers implemented in C can never be broken even if they don't guarantee atomicity (i.e. a dict update making a pure python callback to __hash__ will never result in a broken dict).

This is where Python becomes obscure. Depending on the implementation, a method can be atomic or not :-/ Maybe atomicity should be documented as a "CPython implementation detail"!?

> ISTM that the docs have presumed that people using threading know what they're doing; however, we know that isn't always true ;-)

I don't understand threads and I have no idea which method are thread-safe or not. Each time, I should look at the doc, read the implementation, etc. :-)
msg330130 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-20 13:24
Aha, another strange beast: in Python 3, locale.localeconv() now changes temporarily the LC_CTYPE locale is some cases, and this change affects other threads. That's not something expected.
msg331984 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-17 11:13
About system-wide: see also PR 11190, "time.monotonic() is now always available and always system-wide."
msg402397 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-21 22:22
It would be nice to document thread safety, but it's a large project. Maybe the Documentation Working Group could look into this issue.

But nobody managed to write the doc in 3 years, I prefer to close the issue.
History
Date User Action Args
2021-09-21 22:22:37vstinnersetstatus: open -> closed
resolution: rejected
messages: + msg402397

stage: resolved
2020-08-25 04:59:09sir-sigurdsetnosy: + sir-sigurd
2018-12-17 11:13:29vstinnersetmessages: + msg331984
2018-11-20 13:24:19vstinnersetmessages: + msg330130
2018-11-20 10:21:15vstinnersetmessages: + msg330124
2018-11-19 22:13:05rhettingersetmessages: + msg330117
2018-11-19 19:14:33rhettingersetnosy: + rhettinger
messages: + msg330110
2018-11-19 11:53:08vstinnersetmessages: + msg330095
2018-11-19 11:50:08vstinnercreate