classification
Title: Add --without-decimal-contextvar option to use just threads in decimal
Type: behavior Stage: patch review
Components: Extension Modules Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: skrah Nosy List: boytsovea, gregory.p.smith, mark.dickinson, rhettinger, skrah
Priority: normal Keywords: patch

Created on 2020-02-29 11:24 by skrah, last changed 2020-03-03 15:31 by skrah.

Pull Requests
URL Status Linked Edit
PR 18702 merged skrah, 2020-02-29 17:03
PR 18713 merged skrah, 2020-02-29 20:52
PR 18714 merged skrah, 2020-02-29 21:09
Messages (11)
msg362971 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2020-02-29 11:24
#39776 has shown that it is hard to understand the interaction between
ContextVars and threading in embedded scenarios.

I want to understand the code again, so I'm adding back a compile time
option to enable the thread local context that was present prior to
f13f12d8d.
msg362972 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2020-02-29 11:34
Also, when I'm debugging things like #39776, I don't want to
switch between Python versions.
msg363003 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2020-02-29 18:43
New changeset 815280eb160af637e1347213659f9236adf78f80 by Stefan Krah in branch 'master':
bpo-39794: Add --without-decimal-contextvar (#18702)
https://github.com/python/cpython/commit/815280eb160af637e1347213659f9236adf78f80
msg363016 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2020-02-29 21:16
New changeset 4d7012410cf4f91cbca4c406f4747289c2802333 by Stefan Krah in branch '3.8':
[3.8] bpo-39794: Add --without-decimal-contextvar (GH-18702)
https://github.com/python/cpython/commit/4d7012410cf4f91cbca4c406f4747289c2802333
msg363019 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2020-02-29 22:10
New changeset c4ca1f8f24118dc5c29e16118fb35a13963af290 by Stefan Krah in branch '3.7':
[3.7] bpo-39794: Add --without-decimal-contextvar (GH-18702)
https://github.com/python/cpython/commit/c4ca1f8f24118dc5c29e16118fb35a13963af290
msg363092 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-03-01 20:21
While i don't think this was needed in 3.7 and 3.8, you kept the default behavior the same so I don't think it is a problem.

If someone builds an interpreter with this configure flag, does it break compatibility with anything that code may have started to expect as of 3.7?
msg363111 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-03-02 00:58
+1 for the backport.  A build option isn't new feature, but it does provide a debugging tool and work-around for people who are having problems.
msg363150 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2020-03-02 10:45
For the people who find this issue via a search engine:

  1) The defaults in 3.7, 3.8 and 3.9 are unchanged (you get
     the ContextVar).

  2) ./configure --without-decimal-contextvar enables the exact
     TLS context code from 3.3-3.6.  There is no new code.

  3) On Windows you need to edit PC\pyconfig.h to enable the TLS
     context.

  4) There are extensive tests for the flag in:

        Modules/_decimal/tests/runall-memorydebugger.sh
msg363154 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2020-03-02 11:01
> If someone builds an interpreter with this configure flag, does it break
> compatibility with anything that code may have started to expect as of 3.7?

Yes, anything that relies on the implicit context being coroutine-safe
does not work.  The only test that expectedly breaks in the stdlib is:

   Lib/test/test_asyncio/test_context.py


Basically you can't use operators inside coroutines in the same thread
if both coroutines use a different implicit context.

You can of course replace all operators inside coroutines by the
corresponding context methods:

   # Unsafe with TLS context:
   async def foo():
       with localcontext(Context(prec=10)):
           x = Decimal(1) / 9

   # Safe, just avoid the TLS context:
   async def foo():
       c = Context(prec=10)
       x = c.divide(Decimal(1), 9)
msg363163 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2020-03-02 11:40
To make things more clear (it is good that Gregory asked):

ONLY use this flag if you don't use coroutines at all
OR control ALL async libraries in your application.

Random async libraries from PyPI may rely on the
ContextVar and silently give incorrect results.
msg363268 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2020-03-03 15:31
#39776 is fixed, but _pydecimal was also affected, see msg363266.
History
Date User Action Args
2020-03-03 15:31:20skrahsetmessages: + msg363268
2020-03-02 11:40:49skrahsetmessages: + msg363163
2020-03-02 11:01:22skrahsetmessages: + msg363154
2020-03-02 10:45:59skrahsetmessages: + msg363150
2020-03-02 00:58:32rhettingersetmessages: + msg363111
2020-03-01 20:21:39gregory.p.smithsetnosy: + gregory.p.smith
messages: + msg363092
2020-02-29 22:18:30skrahsetnosy: + boytsovea
2020-02-29 22:10:29skrahsetmessages: + msg363019
2020-02-29 21:16:35skrahsetmessages: + msg363016
2020-02-29 21:09:53skrahsetpull_requests: + pull_request18074
2020-02-29 20:52:34skrahsetpull_requests: + pull_request18072
2020-02-29 18:43:50skrahsetmessages: + msg363003
2020-02-29 17:03:37skrahsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request18064
2020-02-29 11:34:24skrahsetmessages: + msg362972
2020-02-29 11:27:20skrahsetassignee: skrah
stage: needs patch
type: behavior
components: + Extension Modules
versions: + Python 3.7, Python 3.8, Python 3.9
2020-02-29 11:24:13skrahcreate