In versions of Python from 3.4-3.10, the Python core plistlib library supports Apple's binary plist format. When given malformed input, the implementation can be forced to create an argument to struct.unpack() which consumes all available CPU and memory until a MemError is thrown as it builds the 'format' argument to unpack().

This can be seen with the following malformed example binary plist input:

$ xxd binary_plist_dos.bplist
00000000: 6270 6c69 7374 3030 d101 0255 614c 6973  bplist00...UaLis
00000010: 74a5 0304 0506 0000 00df 4251 4351 44a3  t.........BQCQD.
00000020: 0809 0a10 0110 0210 0308 0b11 1719 1b1d  ................
00000030: 0000 0101 0000 0000 0000 000b 0000 0000  ................
00000040: 0000 0000 0000 0000 0000 0029            ...........)

The error is reached in the following lines of plistlib.py:
(

    def _read_ints(self, n, size):
        data = self._fp.read(size * n)
        if size in _BINARY_FORMAT:
            return struct.unpack('>' + _BINARY_FORMAT[size] * n, data)

When the malicious example above is opened by plistlib, it results in 'n' being controlled by the input and it can be forced to be very large. Plistlib attempts to build a string which is as long as 'n', consuming excessive resources.

Apple's built in utilities for handling plist files detects this same file as malformed and will not process it.

Thanks for the report. I can reproduce the issue.

One Apple implementation of binary plist parsing is here: https://opensource.apple.com/source/CF/CF-550/CFBinaryPList.c.

That seems to work from a buffer (or mmap) of the entire file, making consistency checks somewhat easier, and I don't think they have a hardcoded limit on the number of items in the plist (but: it's getting late and might have missed that).

An easy fix for this issue is to limit the amount of values read by _read_ints, but that immediately begs the question what that limit should be. It is clear that 1B items in the array is too much (in this case 1B object refs for dictionary keys).

I don't know what a sane limit would be. The largest plist file on my system is 10MB. Limiting *n* to 20M would be easily enough to parse that file, and doesn't consume too much memory (about 160MB for the read buffer at most).

Another thing the current code doesn't do is check that the "ref" argument to _read_object is in range. That's less problematic because the code will raise an exception regardless (IndexErrox, instead of the nicer InvalidFileException).

There are two issues here.

The simple one is building a large format string for struct.unpack(). It has simple solution: use f'>{n}{_BINARY_FORMAT[size]}'.

The hard issue is that read(n) allocates n bytes in memory even if there are not so many bytes in the file. It affects not only plistlib and should be fixed in the file implementation itself. There is an open issue for this.

Serhiy, thanks. Just the change in the format string would fix this particular example.

I see you're working on a PR with better validation. The current state of the draft looks good to me, but I haven't checked yet if there are other potential problems that can be added to the list of invalid binary plists.

PR 22882 fixes problem in _read_ints(), adds validation for string size, and adds many tests for mailformed binary Plists.

There may be problems with recursive collections. I'll try to solve them too.

No, with recursive collections all is good.

Then I'll just add a NEWS entry and maybe few more tests.

New changeset 34637a0 by Serhiy Storchaka in branch 'master':
bpo-42103: Improve validation of Plist files. (GH-22882)
34637a0

New changeset e277cb7 by Miss Skeleton (bot) in branch '3.9':
bpo-42103: Improve validation of Plist files. (GH-22882)
e277cb7

New changeset 547d2bc by Serhiy Storchaka in branch '3.8':
[3.8] bpo-42103: Improve validation of Plist files. (GH-22882) (GH-23116)
547d2bc

New changeset 225e365 by Serhiy Storchaka in branch '3.7':
[3.7] bpo-42103: Improve validation of Plist files. (GH-22882) (bpo-23117)
225e365

New changeset a63234c by Serhiy Storchaka in branch '3.6':
[3.6] bpo-42103: Improve validation of Plist files. (GH-22882) (GH-23118)
a63234c