This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: copy.copy() does not properly copy os.environment
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: andrei.avk, eryksun, gregory.p.smith, htgoebel, josh.r, kamilturek, methane, mkatsev, pitrou, r.david.murray, rhettinger, rmast, sblondon, serhiy.storchaka
Priority: normal Keywords: needs review, patch

Created on 2012-07-16 16:08 by htgoebel, last changed 2022-04-11 14:57 by admin.

Files
File name Uploaded Description Edit
environcopy.patch anton.barkovsky, 2012-07-16 19:13 Add __copy__ method to os.environ review
environcopy_v2.patch anton.barkovsky, 2012-07-16 19:39 Add __copy__ method to os.environ review
Messages (18)
msg165631 - (view) Author: Hartmut Goebel (htgoebel) Date: 2012-07-16 16:08
Wehn copying os.environ usinf copy.copy(), any manipulation on the copied object will change os.environment, too.

$ python
Python 2.7.3 (default, Apr 22 2012, 07:46:58) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import copy, os
>>> os.environ['TITI']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/UserDict.py", line 23, in __getitem__
    raise KeyError(key)
KeyError: 'TITI'
>>> env = copy.copy(os.environ)
>>> env['TITI'] = 'in den Ferien'
>>> os.environ['TITI']
'in den Ferien'
>>> 

Strictly speaking, this is correct, as the os.environment class is meant to manipulate the environment in the background. But user's expectation is different: he thinks, manipulating the copied object is save and does not effect the environment.
msg165637 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-07-16 16:53
os.environ is not a dictionary, so it isn't all that surprising that a shallow copy doesn't behave like a shallow copy of a dictionary.  deepcopy does what you'd expect, as does os.environ.copy().

Perhaps it is worth improving this by adding a __copy__ method to os.environ.  Someone would need to propose a patch, and it would be an enhancement.
msg165647 - (view) Author: Anton Barkovsky (anton.barkovsky) * Date: 2012-07-16 19:13
Here's a patch.
msg165652 - (view) Author: Anton Barkovsky (anton.barkovsky) * Date: 2012-07-16 19:39
A new patch with tests.
msg165691 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-07-17 08:57
I agree with David that copy(os.environ) is rather ambiguous. I think the preferred idiom should be to call dict(os.environ).
As for __copy__, I don't know what it should do: perhaps simply raise an error when copying is attempted?
msg165712 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-07-17 11:40
Well, I think the fact that os.environ.copy is explicitly supported (and does indeed do dict(os.environ)) is an argument in favor of giving that same meaning to copy(os.environ).  I think that follows the principle of least surprise.
msg286384 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-01-27 21:19
If add support for the copy protocol, consider also adding support for deepcopying and pickling.
msg286392 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-01-28 01:02
[Antoine]
> I agree with David that copy(os.environ) is rather ambiguous.
> I think the preferred idiom should be to call dict(os.environ).
> As for __copy__, I don't know what it should do: perhaps
> simply raise an error when copying is attempted?

FWIW, I concur with Antoine on every point.  Also, raising an error when copying seems reasonable ("refuse the temptation to guess" at what the user intended).
msg286473 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2017-01-30 07:51
patch looks OK.
But I prefer `__deepcopy__ = __copy__ = copy`.

I don't know how to support pickling. (should unpickled object
reference os.environ, or copied dict?)
I feel it is different issue.
msg286513 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-01-31 03:43
That patch would be fine except that I really agree with Antoine that an error should be raised.
msg286526 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2017-01-31 12:51
I agree.
os.environ is not dict, it's a proxy of "current" environment which
is not copyable.
So I agree to copy.copy() shouldn't copy os.environ implicitly.
msg384993 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2021-01-13 04:52
Would we remove the functionality of os.environ.copy()? It seems very odd for types to have a .copy() method that works, while not supporting copy.copy, especially when there is zero documentation, on the web or the docstring, to even hint at the difference.

I'm strongly in favor of silently doing the right thing and behaving the same way the .copy() method already behaves; if you want a "copy" of os.environ that still modifies the environment, that's just aliasing it (envalias = os.environ), not copying at all. If you're trying to make a shallow copy, not an alias, you're trying to separate it from the parent, which every other dict-like thing does (assuming immutable values), where os.environ is a very weird exception (for copy.copy, but not the .copy() method).

Can someone give an example where you'd want copy.copy to produce a "shallow copy" that acts like an alias, not an actual independent copy?
msg396922 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-07-03 20:25
I think it may be good to deprecate and discourage use of `os.environ.copy()`, and add a new method `os.environ.asdict()`. And possibly have `__copy__` raise an error pointing users to `os.environ.asdict()`.

I'm not sure what to do about pickling.
msg399654 - (view) Author: Stéphane Blondon (sblondon) * Date: 2021-08-16 11:51
I think an asdict() method is not necessary. Instanciating a dict from os.environ is enough: 

>>> import os
>>> os.environ
environ({'SHELL': '/bin/bash', ...})
>>> dict(os.environ)
{'SHELL': '/bin/bash', ...})

With 'dict()', it's obvious there is no side effect.


I think os._Environ should not be pickable too because there is an ambiguity too when it will be unpicked: should we set the unpickled data to a basic dict (or another _Environ instance) or to the environment variables?

It seems there are more maintainers in favour of the removal. Removing the copy is a breaking change so it should be done on several releases.

Are you interested by a PR which adds DeprecationWarning when copy.copy(os.environ) or os.environ.copy() are called? It seems to be the first step to implement.
msg402364 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-09-21 20:44
I think the advantage of asdict() method is it's more discoverable and it doesn't leave any uncertainty of whether returned value will update environment or not. If a user sees `dict(os.environ)` in code, they may wonder if it does or does not; and there's no obvious way to confirm.

If they see os.environ.asdict(), it will be safe to assume that the method is provided exactly for this purpose, and the way to confirm is obvious - checking the docstring.
msg402366 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-09-21 21:21
I guess the easy way to test it would be to modify the copy and check os.environ, but still there would be some uncertainty if it's consistent across platforms and python versions, if it's expected to still work in future python versions, etc. asdict() docstring is much more explicit.
msg414318 - (view) Author: Max Katsev (mkatsev) Date: 2022-03-01 23:26
Note that deepcopy doesn't work either, even though it looks like it does at the first glance (which is arguably worse since it's harder to notice):

Python 3.8.6 (default, Jun  4 2021, 05:16:01)
>>> import copy, os, subprocess
>>> env_copy = copy.deepcopy(os.environ)
>>> env_copy["TEST"] = "oh no"
>>> os.environ["TEST"]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/fbcode/platform009/lib/python3.8/os.py", line 675, in __getitem__
    raise KeyError(key) from None
KeyError: 'TEST'
>>> subprocess.run("echo $TEST", shell=True, capture_output=True).stdout.decode()
'oh no\n'
msg414319 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2022-03-02 00:42
In bpo-28824, I suggested preserving the case of environment variables in Windows by using a case-insensitive subclass of str in the encodekey() function. This is self-contained by the use of the encodekey() and decodekey() functions in the mapping methods such as __iter__(). The reason for this is mostly about aesthetics, but also about faithfully displaying the actual environment variable names. Opinions vary, but to me "WindowsSdkVerBinPath" is both easier on my eyes and easier to read than "WINDOWSSDKVERBINPATH". The eyesore factor gets amplified when it's a wall of all upper-cased names 'screaming' at me.

A copy via dict(os.environ) would use regular str keys and lose the case-insensitive, case-preserving property. It would be more useful if os.environ.copy() were implemented to return a copy that keeps the case-insensitive property for keys but disables updating the process environment. This could be implemented by making putenv and unsetenv parameters in the constructor. If self.putenv or self.unsetenv is None, then __setitem__() or __delitem__() woud have no effect on the process environment. The copy() method would return a new _Environ instance for self._data.copy(), one which of course disables updating the process environment.
History
Date User Action Args
2022-04-11 14:57:32adminsetgithub: 59578
2022-03-02 00:42:01eryksunsetnosy: + eryksun
messages: + msg414319
2022-03-01 23:26:40mkatsevsetnosy: + mkatsev
messages: + msg414318
2021-09-21 21:21:58andrei.avksetmessages: + msg402366
2021-09-21 20:51:04anton.barkovskysetnosy: - anton.barkovsky
2021-09-21 20:44:58andrei.avksetmessages: + msg402364
2021-08-18 06:14:42rmastsetnosy: + rmast
2021-08-16 11:51:11sblondonsetnosy: + sblondon

messages: + msg399654
versions: + Python 3.11, - Python 3.10
2021-07-03 20:25:16andrei.avksetnosy: + andrei.avk
messages: + msg396922
2021-03-13 19:09:46kamiltureksetnosy: + kamilturek
2021-01-13 04:52:26josh.rsetnosy: + josh.r
messages: + msg384993
2021-01-12 18:26:21iritkatrielsetcomponents: + Library (Lib)
versions: + Python 3.10, - Python 3.7
2017-01-31 12:51:47methanesetmessages: + msg286526
2017-01-31 03:43:52rhettingersetmessages: + msg286513
2017-01-30 07:51:50methanesetnosy: + methane
messages: + msg286473
2017-01-28 01:02:28rhettingersetnosy: + rhettinger
messages: + msg286392
2017-01-27 21:19:35serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg286384
2017-01-27 21:06:01gregory.p.smithsetkeywords: + needs review
nosy: + gregory.p.smith
stage: patch review

versions: + Python 3.7, - Python 3.4
2012-07-17 11:40:18r.david.murraysetmessages: + msg165712
2012-07-17 08:57:05pitrousetnosy: + pitrou
messages: + msg165691
2012-07-16 19:39:01anton.barkovskysetfiles: + environcopy_v2.patch

messages: + msg165652
2012-07-16 19:13:50anton.barkovskysetfiles: + environcopy.patch

nosy: + anton.barkovsky
messages: + msg165647

keywords: + patch
2012-07-16 16:53:35r.david.murraysetversions: + Python 3.4, - Python 2.7
nosy: + r.david.murray

messages: + msg165637

type: behavior -> enhancement
2012-07-16 16:08:57htgoebelcreate