classification
Title: Document atomic operations on builtin types
Type: enhancement Stage:
Components: Documentation Versions: Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Dima.Tisnek, brett.cannon, docs@python, r.david.murray, rhettinger, terry.reedy
Priority: normal Keywords:

Created on 2015-10-08 15:43 by Dima.Tisnek, last changed 2015-10-09 22:40 by terry.reedy.

Messages (10)
msg252545 - (view) Author: Dima Tisnek (Dima.Tisnek) * Date: 2015-10-08 15:43
Please document what builtin type operations are actually atomic.
For example, what set() operations are atomic?

(There are some blogs / tutorials online, but information is outdated and not authoritative)
msg252559 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-10-08 17:14
We actually don't have any guarantees written down because we have never formalized them. It was discussed at the PyCon language summit this past year -- https://lwn.net/Articles/640177/ -- but it didn't lead to anyone writing a proposal to formalize the memory model.
msg252561 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-10-08 17:29
I wonder...personally I prefer to program in asyncio style rather than threading style, where one doesn't have to worry about atomicity.  Maybe Python shouldn't make any atomicity guarantees.
msg252575 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2015-10-09 00:37
> what set() operations are atomic?

The language doesn't make any guarantees about set operation atomicity.  Different implementations such as PyPy, Jython, and IronPython are free to make different choices than CPython.  In general, users should make no assumptions about atomicity unless explicitly documented and tested.  The wise course of action is to use mutexes when there is any doubt.

FWIW, it is difficult to make blanket statements about the methods on sets because the atomicity depends on the objects looked up or stored in the sets rather than the set itself.   Aside from trivial calls to __sizeof__ and __len__, most set methods potentially call __hash__ or __eq__ on the set elements either of which could make a callback into pure python code.  Likewise, any reference count decrement can potentially make a callback as well.
msg252646 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-10-09 20:23
This question has been asked multiple times before.  I think it should be documented that as far as the language goes, there is no answer.  Raymond's answer is a start.  Dima, where would you expect to find such a disclaimer (other than in the FAQ)?
msg252649 - (view) Author: Dima Tisnek (Dima.Tisnek) * Date: 2015-10-09 20:32
Ideally I'd like 2 sources:

1. a whole section on atomic operations in language and CPython implementation

2. annotation of standard library methods, e.g.:
set().add(element)  [atomic] or [CPython: atomic(*)]
(*) assuming basic types, note about what custom methods would break this guarantee; the reference could be to 1.
msg252651 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-10-09 20:42
I think what Terry was asking was, where would you expect to see the disclaimer that *no* operations are guaranteed to be atomic?  That's what we're inclining toward (though we'll probably need a signoff from Guido).
msg252654 - (view) Author: Dima Tisnek (Dima.Tisnek) * Date: 2015-10-09 20:56
To clarify, Python language disclaimer can be in the general atomic operations or multithreading section.

What I'd really like to see is documented, practical CPython and stdlib behaviour.

I'm under the impression that there's quite a bit of code out there that relies on what's actually atomic in CPython.

d.
msg252665 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-10-09 21:31
I think you are correct, and I wouldn't be surprised if there is some in the stdlib as well.
msg252672 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-10-09 22:40
Yes, I was asking where to put the disclaimer.  The thread docs would be approriate if there is nothing already.

Guido has been very reluctant to put any performance guarantees in the language reference.  I believe he said that O(f(n)) info even for CPython should be in the wiki -- and taken as a guideline, not a iron guarantee.

Further discussion might be better directed to python-ideas, after a search of the pydev and python-ideas archives (most easily done with the gmane mirrors, I believe).
History
Date User Action Args
2015-10-09 22:40:32terry.reedysetmessages: + msg252672
2015-10-09 21:31:07r.david.murraysetmessages: + msg252665
2015-10-09 20:56:57Dima.Tisneksetmessages: + msg252654
2015-10-09 20:42:01r.david.murraysetmessages: + msg252651
2015-10-09 20:32:05Dima.Tisneksetmessages: + msg252649
2015-10-09 20:23:19terry.reedysetnosy: + terry.reedy
messages: + msg252646
2015-10-09 00:37:34rhettingersetnosy: + rhettinger
messages: + msg252575
2015-10-08 17:29:21r.david.murraysetnosy: + r.david.murray
messages: + msg252561
2015-10-08 17:14:46brett.cannonsetnosy: + brett.cannon
messages: + msg252559
2015-10-08 15:43:17Dima.Tisneksetassignee: docs@python

type: enhancement
components: + Documentation
nosy: + docs@python
2015-10-08 15:43:06Dima.Tisnekcreate