classification
Title: Remove usage of tempfile.mktemp in stdlib
Type: security Stage: resolved
Components: Library (Lib), Windows Versions: Python 3.11, Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: dstufft, epaine, eric.araujo, eric.smith, lukasz.langa, miss-islington, paul.moore, serhiy.storchaka, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2020-11-06 14:57 by epaine, last changed 2021-08-29 12:57 by lukasz.langa. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 23200 merged epaine, 2020-11-08 14:57
PR 28024 merged miss-islington, 2021-08-29 11:07
PR 28025 merged miss-islington, 2021-08-29 11:08
PR 28026 merged miss-islington, 2021-08-29 11:08
Messages (11)
msg380450 - (view) Author: E. Paine (epaine) * Date: 2020-11-06 14:57
Currently, there are many uses of `tempfile.mktemp` in the stdlib. I couldn't find an issue where this has already been discussed, but I think the usage of mktemp in the stdlib should be completely reviewed. I grepped the Lib and a slightly filtered version is the following:

Lib/asyncio/windows_utils.py:34: address = tempfile.mktemp(
Lib/distutils/command/bdist_wininst.py:185: archive_basename = mktemp()
Lib/distutils/util.py:386: (script_fd, script_name) = None, mktemp(".py")
Lib/msilib/__init__.py:214: filename = mktemp()
Lib/multiprocessing/connection.py:81: return tempfile.mktemp(prefix='listener-', dir=util.get_temp_dir())
Lib/multiprocessing/connection.py:83: return tempfile.mktemp(prefix=r'\.\pipe\pyc-%d-%d-' %
Lib/pydoc.py:1620: filename = tempfile.mktemp()
Lib/test/bisect_cmd.py:75: tmp = tempfile.mktemp()
Lib/test/test_bytes.py:1193: tfn = tempfile.mktemp()
Lib/test/test_contextlib.py:316: tfn = tempfile.mktemp()
Lib/test/test_doctest.py:2724: >>> fn = tempfile.mktemp()
Lib/test/test_doctest.py:2734: >>> fn = tempfile.mktemp()
Lib/test/test_doctest.py:2744: >>> fn = tempfile.mktemp()
Lib/test/test_faulthandler.py:51: filename = tempfile.mktemp()
Lib/test/test_shutil.py:1624: filename = tempfile.mktemp(dir=dirname)
Lib/test/test_shutil.py:1935: dst_dir = tempfile.mktemp(dir=self.mkdtemp())
Lib/test/test_shutil.py:2309: name = tempfile.mktemp(dir=os.getcwd())
Lib/test/test_shutil.py:272: filename = tempfile.mktemp(dir=self.mkdtemp())
Lib/test/test_shutil.py:677: dst = tempfile.mktemp(dir=self.mkdtemp())
Lib/test/test_socket.py:699: path = tempfile.mktemp(dir=self.dir_path)
Lib/test/test_socketserver.py:100: fn = tempfile.mktemp(prefix='unix_socket.', dir=dir)

I am hoping this issue will be spotted as I couldn't find who to add to the nosy for this. I think, bearing in mind that use of this method is a security issue, we should reduce this number as low as feasible (though, I am sure that a number of those will have good reasons for using mktemp, and will be doing so in a safe way).
msg380535 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-11-08 07:23
Most of them are in tests. There is no security issue there, also the code may be clearer and more reliable if use helper function test.support.temp_dir().

And most of the rest are in Windows specific code. Some Windows code may not work if you hold open file descriptor, so we should ensure that that code is tested.
msg380537 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-11-08 08:22
Yeah, once tests are excluded and the (deprecated or nearly deprecated) distutils and msilib are dropped, the problems are pydoc (which looks non-exploitable) and anywhere we need to generate a named pipe.

Both cases where named pipes are being created are as safe as the OS allows, so it's really just pydoc that might deserve a fix. (For reference, it's in the variation of help() that writes the docstring to a file and triggers the equivalent of "type <file> | more" or "cat <file> | less", which is already only useful in an interactive shell.)

So I'd suggest it's already as low as possible, but if someone wants to fix pydoc (and encourage the SC to approve PEP 594 and PEP 632 so we don't have to worry about msilib or distutils) then they can feel free.
msg380553 - (view) Author: E. Paine (epaine) * Date: 2020-11-08 14:41
> Most of them are in tests. There is no security issue there
TBH, I don't know enough about the exploit to comment, but it seems that the tempfile tests take this seriously (Lib/test/test_tempfile.py:782 "For safety, all use of mktemp must occur in a private directory.")

> distutils and msilib are dropped
Is this wise? As you noted, PEP 594 and PEP 632 have yet to be approved (in which case, should we not still be looking at these modules, particularly as PEP 594 has been around for a while).

> if someone wants to fix pydoc

I am currently drafting a PR which will replace it with `NamedTemporaryFile` (and while we're at it, replace the `os.system` call with `subprocess.run`)
msg381168 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-11-16 21:45
Just left a blocking review on the PR - I don't want to rely on the shell being able to use an already open file.

There's at least one other issue about making NamedTemporaryFile work for this case. Once that is done, this can be made to work.
msg386232 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-02-03 18:03
Distutils is now deprecated (see PEP 632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools
msg400524 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-08-29 11:07
New changeset c9227df5a9d8e958a2324cf0deba8524d1ded26a by E-Paine in branch 'main':
bpo-42278: Use tempfile.TemporaryDirectory rather than tempfile.mktemp in pydoc (GH-23200)
https://github.com/python/cpython/commit/c9227df5a9d8e958a2324cf0deba8524d1ded26a
msg400529 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-08-29 12:56
New changeset 45409518c1cec5ee91d49f69a2f8eb4196d242f0 by Miss Islington (bot) in branch '3.9':
bpo-42278: Use tempfile.TemporaryDirectory rather than tempfile.mktemp in pydoc (GH-23200) (GH-28025)
https://github.com/python/cpython/commit/45409518c1cec5ee91d49f69a2f8eb4196d242f0
msg400530 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-08-29 12:57
New changeset 193443bb708cba3a72e99e61dd6615a94f22f9e1 by Miss Islington (bot) in branch '3.8':
bpo-42278: Use tempfile.TemporaryDirectory rather than tempfile.mktemp in pydoc (GH-23200) (GH-28026)
https://github.com/python/cpython/commit/193443bb708cba3a72e99e61dd6615a94f22f9e1
msg400531 - (view) Author: miss-islington (miss-islington) Date: 2021-08-29 12:57
New changeset 532ebba6c8697d214a0d94514ad0b2464a59cb7c by Miss Islington (bot) in branch '3.10':
bpo-42278: Use tempfile.TemporaryDirectory rather than tempfile.mktemp in pydoc (GH-23200)
https://github.com/python/cpython/commit/532ebba6c8697d214a0d94514ad0b2464a59cb7c
msg400532 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-08-29 12:57
Thanks, E. Paine! ✨ 🍰 ✨
History
Date User Action Args
2021-08-29 12:57:59lukasz.langasetstatus: open -> closed
versions: + Python 3.11, - Python 3.7
messages: + msg400532

resolution: fixed
stage: patch review -> resolved
2021-08-29 12:57:26miss-islingtonsetmessages: + msg400531
2021-08-29 12:57:09lukasz.langasetmessages: + msg400530
2021-08-29 12:56:49lukasz.langasetmessages: + msg400529
2021-08-29 11:08:10miss-islingtonsetpull_requests: + pull_request26472
2021-08-29 11:08:05miss-islingtonsetpull_requests: + pull_request26471
2021-08-29 11:07:59miss-islingtonsetnosy: + miss-islington

pull_requests: + pull_request26470
stage: resolved -> patch review
2021-08-29 11:07:55lukasz.langasetnosy: + lukasz.langa
messages: + msg400524
2021-02-03 20:36:03epainesetstatus: closed -> open
resolution: out of date -> (no value)
components: - Distutils
2021-02-03 18:03:25steve.dowersetstatus: open -> closed
resolution: out of date
messages: + msg386232

stage: patch review -> resolved
2020-11-16 21:45:00steve.dowersetmessages: + msg381168
2020-11-08 14:57:40epainesetkeywords: + patch
stage: patch review
pull_requests: + pull_request22100
2020-11-08 14:41:33epainesetmessages: + msg380553
2020-11-08 08:22:36steve.dowersetmessages: + msg380537
2020-11-08 07:23:33serhiy.storchakasetnosy: + paul.moore, tim.golden, serhiy.storchaka, zach.ware, steve.dower
messages: + msg380535
components: + Windows
2020-11-07 17:37:48epainesetnosy: + eric.araujo, dstufft
components: + Distutils
2020-11-06 15:35:27eric.smithsetnosy: + eric.smith
2020-11-06 14:57:52epainecreate