classification
Title: Document os.environ[x] = y and os.putenv() as thread unsafe
Type: Stage:
Components: Documentation Versions: Python 3.9, Python 3.8, Python 3.7, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Daniel Martin, docs@python, eryksun, gregory.p.smith, nanjekyejoannah
Priority: normal Keywords:

Created on 2020-01-18 00:36 by gregory.p.smith, last changed 2020-08-27 13:40 by Daniel Martin.

Messages (5)
msg360221 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-01-18 00:36
The underlying API calls made by os.putenv() and os.environ[name] = value syntax are not thread safe on POSIX systems.  POSIX _does not have_ any thread safe way to access the process global environment.

In a pure Python program, the GIL prevents this from being an issue.  But when embedded in a C/C++ program or using extension modules that launch their own threads from C, those threads could also make the invalid assumption that they can safely _read_ the environment.  Which is a race condition when a Python thread is doing a putenv() at the same time.

We should document the danger.

CPython's os module snapshots a copy of the environment into a dict at import time (during CPython startup).  But os.environ[] assignment and os.putenv() modify the actual process global environment in addition to updating this dict.  (If an embedded interpreter is launched from a process with other threads already running, even that initial environment reading could be unsafe if the larger application has a thread that wrongly assumes it has exclusive environment access)

For people modifying os.environ so that the change is visible to child processes, we can recommend using the env= parameter on subprocess API calls to supply a new environment.

A broader issue of should we be modifying the process global environment state at all from os.putenv() and os.environ[] assignment still exists.  I'll track that in another issue (to be opened).
msg360223 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-01-18 00:47
https://bugs.python.org/issue39376 tracks possible interpreter behavior changes.
msg360450 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-01-22 06:35
fwiw, no need to remove that message.  We'll want to make the docs clear that this does not apply to Windows.  :)
msg360451 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-01-22 08:11
> no need to remove that message.  

I was discussing the wrong API. It's not directly relevant that Windows API functions that access the process environment are protected by the PEB lock. Python primarily uses [_w]environ, a copy of the process environment that's managed by the C runtime library (ucrt). os.putenv modifies this environment via _wputenv. (Ultimately this syncs with the process environment by calling SetEnvironmentVariableW.)

Functions that modify and read ucrt's environment are protected by a lock. But there's still a concern if multithreaded code reads or modifies [_w]environ concurrent to a _[w]putenv call. Also, [_w]getenv returns a pointer to the value in [_w]environ, so it has the same problem. A significant difference, however, is that _[w]putenv in ucrt is not POSIX compliant, since it copies the caller's string. Also, ucrt has safer [_w]getenv_s and _[w]dupenv_s functions that return a copy.
msg375992 - (view) Author: Daniel Martin (Daniel Martin) Date: 2020-08-27 13:40
See also https://bugs.python.org/issue40961 - that bug is not about thread safety, but another quirk around env. variable setting that needs to be documented in the documentation for os.putenv and os.getenv, and so anyone addressing this bug should probably address that one at the same time.
History
Date User Action Args
2020-08-27 13:40:07Daniel Martinsetnosy: + Daniel Martin
messages: + msg375992
2020-07-06 20:00:05nanjekyejoannahsetnosy: + nanjekyejoannah
2020-01-22 08:11:06eryksunsetmessages: + msg360451
2020-01-22 06:35:21gregory.p.smithsetmessages: + msg360450
2020-01-22 00:50:05eryksunsetmessages: - msg360245
2020-01-18 14:07:02eryksunsetnosy: + eryksun
messages: + msg360245
2020-01-18 00:47:59gregory.p.smithsetmessages: + msg360223
2020-01-18 00:36:08gregory.p.smithcreate