classification
Title: Pickle crashes unpickling invalid NEWOBJ_EX opcode
Type: crash Stage: resolved
Components: Extension Modules Versions: Python 3.10, Python 3.9, Python 3.8, Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Iman Sharafaldin, belopolsky, christian.heimes, lukasz.langa, miss-islington, ned.deily, p-ganssle, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2020-07-12 21:36 by Iman Sharafaldin, last changed 2020-07-26 15:12 by Iman Sharafaldin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 21458 merged serhiy.storchaka, 2020-07-13 12:11
PR 21459 merged miss-islington, 2020-07-13 12:50
PR 21460 merged miss-islington, 2020-07-13 12:50
PR 21461 merged miss-islington, 2020-07-13 13:08
PR 21462 merged miss-islington, 2020-07-13 13:08
PR 21472 merged serhiy.storchaka, 2020-07-14 12:07
Messages (15)
msg373573 - (view) Author: Iman Sharafaldin (Iman Sharafaldin) Date: 2020-07-12 21:36
The following code generates a segfault on the Pickle module [it's a crafted datetime object] (Python 3.10.0a0 (heads/master:b40e434, Jul  4 2020), Python 3.6.11 and Python 3.7.2):

import io
import pickle


hex_string = "8004952A000000000000008C086461746574696D65948C086461746574696D65949388430A07B2010100000000000092059452942E"
myb = bytes.fromhex(hex_string)
f = io.BytesIO(myb)
print(f)
data = pickle.load(f)
print(data)
print('We have segfault but we cannot see!')
msg373575 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-07-12 21:38
datetime_new assumes that args is a tuple. load_newobj_ex() doesn't check that args is a tuple and kwargs is a dictionary. The demo exploit passes ``True`` as args, which triggers a segfault in PyTuple_GET_SIZE in datetime_new.

#0  datetime_new (type=0x7fffea5d2740 <PyDateTime_DateTimeType>, args=True, kw=b'\x07\xb2\x01\x01\x00\x00\x00\x00\x00\x00') at /usr/src/debug/python3-3.8.3-2.fc32.x86_64/Modules/_datetimemodule.c:4737
#1  0x00007fffea637b1e in load_newobj_ex (self=0x7fffea7a7820) at /usr/src/debug/python3-3.8.3-2.fc32.x86_64/Modules/_pickle.c:6008
#2  0x00007fffea632e7a in load (self=0x7fffea7a7820) at /usr/src/debug/python3-3.8.3-2.fc32.x86_64/Modules/_pickle.c:6943
#3  0x00007fffea63795e in _pickle_load_impl (module=<optimized out>, buffers=0x0, errors=0x7fffea639149 "strict", encoding=0x7fffea6391fa "ASCII", fix_imports=1, file=<_io.BytesIO at remote 0x7fffea888180>)
    at /usr/src/debug/python3-3.8.3-2.fc32.x86_64/Modules/_pickle.c:1688
#4  _pickle_load (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3-3.8.3-2.fc32.x86_64/Modules/clinic/_pickle.c.h:731
#5  0x00007ffff7bd1ced in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<built-in method load of module object at remote 0x7fffea680270>, args=<optimized out>, nargsf=<optimized out>, kwnames=0x0)
    at /usr/src/debug/python3-3.8.3-2.fc32.x86_64/Objects/methodobject.c:437
msg373599 - (view) Author: miss-islington (miss-islington) Date: 2020-07-13 12:49
New changeset 4f309abf55f0e6f8950ac13d6ec83c22b8d47bf8 by Serhiy Storchaka in branch 'master':
bpo-41288: Fix a crash in unpickling invalid NEWOBJ_EX. (GH-21458)
https://github.com/python/cpython/commit/4f309abf55f0e6f8950ac13d6ec83c22b8d47bf8
msg373600 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-07-13 12:51
Ned, would you like to get this fix backported to 3.6 and 3.7?
msg373603 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-07-13 13:05
Sounds like a good idea.
msg373604 - (view) Author: miss-islington (miss-islington) Date: 2020-07-13 13:05
New changeset f56c75ed53dcad4d59dff4377ae463d6b96acd3e by Miss Islington (bot) in branch '3.8':
bpo-41288: Fix a crash in unpickling invalid NEWOBJ_EX. (GH-21458)
https://github.com/python/cpython/commit/f56c75ed53dcad4d59dff4377ae463d6b96acd3e
msg373605 - (view) Author: miss-islington (miss-islington) Date: 2020-07-13 13:09
New changeset 57c984fab69b862563899d2c11da7d27201f4152 by Miss Islington (bot) in branch '3.9':
bpo-41288: Fix a crash in unpickling invalid NEWOBJ_EX. (GH-21458)
https://github.com/python/cpython/commit/57c984fab69b862563899d2c11da7d27201f4152
msg373616 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-07-13 18:17
New changeset 620e276a8c1d53332fbf08d369be87f862b6949d by Miss Islington (bot) in branch '3.7':
bpo-41288: Fix a crash in unpickling invalid NEWOBJ_EX. (GH-21458) (GH-21461)
https://github.com/python/cpython/commit/620e276a8c1d53332fbf08d369be87f862b6949d
msg373617 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-07-13 18:18
New changeset 6463cf07fef7a923a743fcaf312150c45fd81b64 by Miss Islington (bot) in branch '3.6':
bpo-41288: Fix a crash in unpickling invalid NEWOBJ_EX. (GH-21458) (GH-21462)
https://github.com/python/cpython/commit/6463cf07fef7a923a743fcaf312150c45fd81b64
msg373886 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-07-18 08:11
New changeset b4c98ed41e6c959e95b2a6f65c1b728e94039dfd by Serhiy Storchaka in branch 'master':
bpo-41288: Refactor of unpickling NEWOBJ and NEWOBJ_EX opcodes. (GH-21472)
https://github.com/python/cpython/commit/b4c98ed41e6c959e95b2a6f65c1b728e94039dfd
msg374022 - (view) Author: Ɓukasz Langa (lukasz.langa) * (Python committer) Date: 2020-07-20 17:29
Is this fix causing the refleak build to fail on 3.8? See: https://buildbot.python.org/all/#/builders/484/builds/149
msg374086 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-07-22 05:32
This fix touched the unpickling code, and the test crashed during pickling, so it is not directly related. But it may be some compiler glitch, when changing one part of code affects compilation of other parts of code. The next run on the buildbot was passed, and there were failures in the past (but logs were not saved), so it may be just unstable test.
msg374311 - (view) Author: Iman Sharafaldin (Iman Sharafaldin) Date: 2020-07-26 13:32
@serhiy.storchaka Hi Serhiy, do you count this issue as a security issue? Also, I didn't see that in the latest changelog. Is it now patched on the latest version?

https://docs.python.org/release/3.8.5/whatsnew/changelog.html
msg374314 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-07-26 14:19
I do not think it is a security issue. The crash cannot be triggered by the user input unless you accept the pickle data from untrusted sources, but in that case you are in large danger, because you allow executing arbitrary code.

The changes in this issue just help to debug in some cases when you play with pickle format. In any case thank you for your report.
msg374320 - (view) Author: Iman Sharafaldin (Iman Sharafaldin) Date: 2020-07-26 15:12
Thank you for patching that.
History
Date User Action Args
2020-08-03 17:29:14vstinnerlinkissue41208 superseder
2020-07-26 15:12:33Iman Sharafaldinsetmessages: + msg374320
2020-07-26 14:19:49serhiy.storchakasetstatus: open -> closed

messages: + msg374314
2020-07-26 13:32:40Iman Sharafaldinsetmessages: + msg374311
2020-07-22 05:32:06serhiy.storchakasetmessages: + msg374086
2020-07-21 19:31:48serhiy.storchakasetstatus: closed -> open
2020-07-20 17:29:59lukasz.langasetnosy: + lukasz.langa
messages: + msg374022
2020-07-18 08:11:25serhiy.storchakasetmessages: + msg373886
2020-07-14 12:07:48serhiy.storchakasetpull_requests: + pull_request20616
2020-07-13 18:19:55ned.deilysetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-07-13 18:18:11ned.deilysetmessages: + msg373617
2020-07-13 18:17:09ned.deilysetmessages: + msg373616
2020-07-13 13:09:30miss-islingtonsetmessages: + msg373605
2020-07-13 13:08:45miss-islingtonsetpull_requests: + pull_request20609
2020-07-13 13:08:32miss-islingtonsetpull_requests: + pull_request20608
2020-07-13 13:05:51miss-islingtonsetmessages: + msg373604
2020-07-13 13:05:25ned.deilysetmessages: + msg373603
versions: + Python 3.6, Python 3.7
2020-07-13 12:51:23christian.heimessetnosy: + ned.deily

messages: + msg373600
versions: + Python 3.8, Python 3.9
2020-07-13 12:50:24miss-islingtonsetpull_requests: + pull_request20607
2020-07-13 12:50:15miss-islingtonsetpull_requests: + pull_request20606
2020-07-13 12:49:52miss-islingtonsetnosy: + miss-islington
messages: + msg373599
2020-07-13 12:47:54christian.heimessettitle: Pickle crashes unpickling invalig NEWOBJ_EX opcode -> Pickle crashes unpickling invalid NEWOBJ_EX opcode
2020-07-13 12:29:45serhiy.storchakasettitle: Pickle crashes using a crafted datetime object -> Pickle crashes unpickling invalig NEWOBJ_EX opcode
2020-07-13 12:11:24serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request20605
2020-07-13 02:25:24xtreaksetnosy: + belopolsky, p-ganssle
2020-07-12 22:00:02serhiy.storchakasetassignee: serhiy.storchaka

components: + Extension Modules, - Interpreter Core
nosy: + serhiy.storchaka
2020-07-12 21:38:55christian.heimessetnosy: + christian.heimes
messages: + msg373575
2020-07-12 21:36:09Iman Sharafaldincreate