Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove usage of tempfile.mktemp in stdlib #86444

Closed
E-Paine mannequin opened this issue Nov 6, 2020 · 11 comments
Closed

Remove usage of tempfile.mktemp in stdlib #86444

E-Paine mannequin opened this issue Nov 6, 2020 · 11 comments
Labels
3.8 only security fixes 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes OS-windows stdlib Python modules in the Lib dir type-security A security issue

Comments

@E-Paine
Copy link
Mannequin

E-Paine mannequin commented Nov 6, 2020

BPO 42278
Nosy @pfmoore, @ericvsmith, @tjguk, @merwok, @ambv, @zware, @serhiy-storchaka, @zooba, @dstufft, @miss-islington, @E-Paine
PRs
  • bpo-42278: Use tempfile.TemporaryDirectory rather than tempfile.mktemp in pydoc #23200
  • [3.10] bpo-42278: Use tempfile.TemporaryDirectory rather than tempfile.mktemp in pydoc (GH-23200) #28024
  • [3.9] bpo-42278: Use tempfile.TemporaryDirectory rather than tempfile.mktemp in pydoc (GH-23200) #28025
  • [3.8] bpo-42278: Use tempfile.TemporaryDirectory rather than tempfile.mktemp in pydoc (GH-23200) #28026
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-08-29.12:57:59.497>
    created_at = <Date 2020-11-06.14:57:52.124>
    labels = ['type-security', '3.8', '3.9', '3.10', '3.11', 'library', 'OS-windows']
    title = 'Remove usage of tempfile.mktemp in stdlib'
    updated_at = <Date 2021-08-29.12:57:59.496>
    user = 'https://github.com/E-Paine'

    bugs.python.org fields:

    activity = <Date 2021-08-29.12:57:59.496>
    actor = 'lukasz.langa'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-08-29.12:57:59.497>
    closer = 'lukasz.langa'
    components = ['Library (Lib)', 'Windows']
    creation = <Date 2020-11-06.14:57:52.124>
    creator = 'epaine'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 42278
    keywords = ['patch']
    message_count = 11.0
    messages = ['380450', '380535', '380537', '380553', '381168', '386232', '400524', '400529', '400530', '400531', '400532']
    nosy_count = 11.0
    nosy_names = ['paul.moore', 'eric.smith', 'tim.golden', 'eric.araujo', 'lukasz.langa', 'zach.ware', 'serhiy.storchaka', 'steve.dower', 'dstufft', 'miss-islington', 'epaine']
    pr_nums = ['23200', '28024', '28025', '28026']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'security'
    url = 'https://bugs.python.org/issue42278'
    versions = ['Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

    @E-Paine
    Copy link
    Mannequin Author

    E-Paine mannequin commented Nov 6, 2020

    Currently, there are many uses of tempfile.mktemp in the stdlib. I couldn't find an issue where this has already been discussed, but I think the usage of mktemp in the stdlib should be completely reviewed. I grepped the Lib and a slightly filtered version is the following:

    Lib/asyncio/windows_utils.py:34: address = tempfile.mktemp(
    Lib/distutils/command/bdist_wininst.py:185: archive_basename = mktemp()
    Lib/distutils/util.py:386: (script_fd, script_name) = None, mktemp(".py")
    Lib/msilib/__init__.py:214: filename = mktemp()
    Lib/multiprocessing/connection.py:81: return tempfile.mktemp(prefix='listener-', dir=util.get_temp_dir())
    Lib/multiprocessing/connection.py:83: return tempfile.mktemp(prefix=r'\.\pipe\pyc-%d-%d-' %
    Lib/pydoc.py:1620: filename = tempfile.mktemp()
    Lib/test/bisect_cmd.py:75: tmp = tempfile.mktemp()
    Lib/test/test_bytes.py:1193: tfn = tempfile.mktemp()
    Lib/test/test_contextlib.py:316: tfn = tempfile.mktemp()
    Lib/test/test_doctest.py:2724: >>> fn = tempfile.mktemp()
    Lib/test/test_doctest.py:2734: >>> fn = tempfile.mktemp()
    Lib/test/test_doctest.py:2744: >>> fn = tempfile.mktemp()
    Lib/test/test_faulthandler.py:51: filename = tempfile.mktemp()
    Lib/test/test_shutil.py:1624: filename = tempfile.mktemp(dir=dirname)
    Lib/test/test_shutil.py:1935: dst_dir = tempfile.mktemp(dir=self.mkdtemp())
    Lib/test/test_shutil.py:2309: name = tempfile.mktemp(dir=os.getcwd())
    Lib/test/test_shutil.py:272: filename = tempfile.mktemp(dir=self.mkdtemp())
    Lib/test/test_shutil.py:677: dst = tempfile.mktemp(dir=self.mkdtemp())
    Lib/test/test_socket.py:699: path = tempfile.mktemp(dir=self.dir_path)
    Lib/test/test_socketserver.py:100: fn = tempfile.mktemp(prefix='unix_socket.', dir=dir)

    I am hoping this issue will be spotted as I couldn't find who to add to the nosy for this. I think, bearing in mind that use of this method is a security issue, we should reduce this number as low as feasible (though, I am sure that a number of those will have good reasons for using mktemp, and will be doing so in a safe way).

    @E-Paine E-Paine mannequin added 3.7 (EOL) end of life 3.10 only security fixes 3.8 only security fixes 3.9 only security fixes stdlib Python modules in the Lib dir type-security A security issue labels Nov 6, 2020
    @serhiy-storchaka
    Copy link
    Member

    Most of them are in tests. There is no security issue there, also the code may be clearer and more reliable if use helper function test.support.temp_dir().

    And most of the rest are in Windows specific code. Some Windows code may not work if you hold open file descriptor, so we should ensure that that code is tested.

    @zooba
    Copy link
    Member

    zooba commented Nov 8, 2020

    Yeah, once tests are excluded and the (deprecated or nearly deprecated) distutils and msilib are dropped, the problems are pydoc (which looks non-exploitable) and anywhere we need to generate a named pipe.

    Both cases where named pipes are being created are as safe as the OS allows, so it's really just pydoc that might deserve a fix. (For reference, it's in the variation of help() that writes the docstring to a file and triggers the equivalent of "type <file> | more" or "cat <file> | less", which is already only useful in an interactive shell.)

    So I'd suggest it's already as low as possible, but if someone wants to fix pydoc (and encourage the SC to approve PEP-594 and PEP-632 so we don't have to worry about msilib or distutils) then they can feel free.

    @E-Paine
    Copy link
    Mannequin Author

    E-Paine mannequin commented Nov 8, 2020

    Most of them are in tests. There is no security issue there
    TBH, I don't know enough about the exploit to comment, but it seems that the tempfile tests take this seriously (Lib/test/test_tempfile.py:782 "For safety, all use of mktemp must occur in a private directory.")

    distutils and msilib are dropped
    Is this wise? As you noted, PEP-594 and PEP-632 have yet to be approved (in which case, should we not still be looking at these modules, particularly as PEP-594 has been around for a while).

    if someone wants to fix pydoc

    I am currently drafting a PR which will replace it with NamedTemporaryFile (and while we're at it, replace the os.system call with subprocess.run)

    @zooba
    Copy link
    Member

    zooba commented Nov 16, 2020

    Just left a blocking review on the PR - I don't want to rely on the shell being able to use an already open file.

    There's at least one other issue about making NamedTemporaryFile work for this case. Once that is done, this can be made to work.

    @zooba
    Copy link
    Member

    zooba commented Feb 3, 2021

    Distutils is now deprecated (see PEP-632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

    If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools

    @zooba zooba closed this as completed Feb 3, 2021
    @zooba zooba closed this as completed Feb 3, 2021
    @E-Paine E-Paine mannequin removed the stdlib Python modules in the Lib dir label Feb 3, 2021
    @E-Paine E-Paine mannequin reopened this Feb 3, 2021
    @E-Paine E-Paine mannequin removed the stdlib Python modules in the Lib dir label Feb 3, 2021
    @E-Paine E-Paine mannequin reopened this Feb 3, 2021
    @ambv
    Copy link
    Contributor

    ambv commented Aug 29, 2021

    New changeset c9227df by E-Paine in branch 'main':
    bpo-42278: Use tempfile.TemporaryDirectory rather than tempfile.mktemp in pydoc (GH-23200)
    c9227df

    @ambv
    Copy link
    Contributor

    ambv commented Aug 29, 2021

    New changeset 4540951 by Miss Islington (bot) in branch '3.9':
    bpo-42278: Use tempfile.TemporaryDirectory rather than tempfile.mktemp in pydoc (GH-23200) (GH-28025)
    4540951

    @ambv
    Copy link
    Contributor

    ambv commented Aug 29, 2021

    New changeset 193443b by Miss Islington (bot) in branch '3.8':
    bpo-42278: Use tempfile.TemporaryDirectory rather than tempfile.mktemp in pydoc (GH-23200) (GH-28026)
    193443b

    @miss-islington
    Copy link
    Contributor

    New changeset 532ebba by Miss Islington (bot) in branch '3.10':
    bpo-42278: Use tempfile.TemporaryDirectory rather than tempfile.mktemp in pydoc (GH-23200)
    532ebba

    @ambv
    Copy link
    Contributor

    ambv commented Aug 29, 2021

    Thanks, E. Paine! ✨ 🍰 ✨

    @ambv ambv added 3.11 only security fixes and removed 3.7 (EOL) end of life labels Aug 29, 2021
    @ambv ambv closed this as completed Aug 29, 2021
    @ambv ambv added 3.11 only security fixes and removed 3.7 (EOL) end of life labels Aug 29, 2021
    @ambv ambv closed this as completed Aug 29, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes OS-windows stdlib Python modules in the Lib dir type-security A security issue
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants