classification
Title: plistlib unable to load iOS7 Safari History.plist
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.5, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: ned.deily, python-dev, ronaldoussoren, serhiy.storchaka, slo.sleuth
Priority: normal Keywords: patch

Created on 2014-05-19 22:08 by slo.sleuth, last changed 2014-05-23 14:09 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
History.plist slo.sleuth, 2014-05-20 04:41
plistlib_read_refs_2.patch serhiy.storchaka, 2014-05-20 06:07 review
Messages (13)
msg218821 - (view) Author: John Lehr (slo.sleuth) Date: 2014-05-19 22:08
plistlib has key error when attempting to load the iOS7  
var/mobile/Applications/com.apple.mobilesafari/Library/Safari/History.plist.  No detected issues with other iOS7 plists.

In [8]: with open('cases/vitale_test_iphone/unback/AppDomain-com.apple.mobilesafari/Library/Safari/History.plist', 'rb') as f:
   ...:     plist = plistlib.lo
plistlib.load   plistlib.loads  
   ...:     plist = plistlib.load(f)
   ...:     
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-8-7824ac03848c> in <module>()
      1 with open('cases/vitale_test_iphone/unback/AppDomain-com.apple.mobilesafari/Library/Safari/History.plist', 'rb') as f:
----> 2     plist = plistlib.load(f)
      3 

/usr/lib/python3.4/plistlib.py in load(fp, fmt, use_builtin_types, dict_type)
    993         p = _FORMATS[fmt]['parser'](use_builtin_types=use_builtin_types)
    994 
--> 995     return p.parse(fp)
    996 
    997 

/usr/lib/python3.4/plistlib.py in parse(self, fp)
    620             ) = struct.unpack('>6xBBQQQ', trailer)
    621             self._fp.seek(offset_table_offset)
--> 622             offset_format = '>' + _BINARY_FORMAT[offset_size] * num_objects
    623             self._ref_format = _BINARY_FORMAT[self._ref_size]
    624             self._object_offsets = struct.unpack(

KeyError: 3
msg218824 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-05-19 23:18
Can you attach a zipped version of the failing plist or at least identify what data in it is causing the exception?  Presumably that plist is user-specific, i.e. contains browser history.
msg218825 - (view) Author: John Lehr (slo.sleuth) Date: 2014-05-20 00:09
I have shared the file for which I have permission, but neither of two
History.plists in my possession will load, both with the same traceback.

Thank you.

On Mon, May 19, 2014 at 4:18 PM, Ned Deily <report@bugs.python.org> wrote:

>
> Ned Deily added the comment:
>
> Can you attach a zipped version of the failing plist or at least identify
> what data in it is causing the exception?  Presumably that plist is
> user-specific, i.e. contains browser history.
>
> ----------
> nosy: +ned.deily
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue21538>
> _______________________________________
>
msg218827 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-05-20 00:51
John, a quick look at the file shows that it contains browsing history information that might be personally identifiable.  I don't think that is appropriate to store on a public web site like this one so I've deleted the file.  Is there some way to provide a simpler file with non-personal history?
msg218834 - (view) Author: John Lehr (slo.sleuth) Date: 2014-05-20 04:41
Ned, I understand.  I did get permission to post that history, and as I
feared, the sanitized History.plist I'm attaching now doesn't have the
loading issue that the first did.  Possibly a URL character is throwing the
exception?

Can you suggest where I can put a print statement in the library or some
other way to post to get you meaningful debug information?  I don't have
any experience with pdb, and a verbose python execution is pretty lengthy.
 As I said before, I have two more extensive History.plist files from two
different devices that won't load.

On Mon, May 19, 2014 at 5:51 PM, Ned Deily <report@bugs.python.org> wrote:

>
> Ned Deily added the comment:
>
> John, a quick look at the file shows that it contains browsing history
> information that might be personally identifiable.  I don't think that is
> appropriate to store on a public web site like this one so I've deleted the
> file.  Is there some way to provide a simpler file with non-personal
> history?
>
> ----------
> nosy: +ned.deily
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue21538>
> _______________________________________
>
msg218835 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-05-20 05:32
I think this patch should fix the issue. But tests needed.
msg218838 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-05-20 06:07
And here is a patch with test.
msg218850 - (view) Author: John Lehr (slo.sleuth) Date: 2014-05-20 16:58
Serhiy, I applied the patch and can confirm this is a fix (three previous
failing loads of History.plists from three different devices now load
successfully).  Thank you, I would not likely have identified the issue
myself.

On Mon, May 19, 2014 at 11:08 PM, Serhiy Storchaka
<report@bugs.python.org>wrote:

>
> Changes by Serhiy Storchaka <storchaka@gmail.com>:
>
>
> Removed file: http://bugs.python.org/file35299/plistlib_read_refs.patch
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue21538>
> _______________________________________
>
msg218854 - (view) Author: John Lehr (slo.sleuth) Date: 2014-05-20 17:31
Patch plistlib_read_refs_2.patch corrects load error for iOS Safari History.plist.
msg218855 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-05-20 18:04
Issue shouldn't be closed as resolved until patch is committed.

Ronald, is it good to you?
msg218878 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2014-05-21 15:02
The patch looks good.
msg218968 - (view) Author: Roundup Robot (python-dev) Date: 2014-05-23 13:15
New changeset f0452bc62cc3 by Serhiy Storchaka in branch '3.4':
Issue #21538: The plistlib module now supports loading of binary plist files
http://hg.python.org/cpython/rev/f0452bc62cc3

New changeset b2c5d0cba5fd by Serhiy Storchaka in branch 'default':
Issue #21538: The plistlib module now supports loading of binary plist files
http://hg.python.org/cpython/rev/b2c5d0cba5fd
msg218971 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-05-23 13:26
Thank you John for your report.
History
Date User Action Args
2014-05-23 14:09:46serhiy.storchakasetstatus: open -> closed
assignee: serhiy.storchaka
resolution: fixed
stage: patch review -> resolved
2014-05-23 13:26:27serhiy.storchakasetmessages: + msg218971
2014-05-23 13:15:58python-devsetnosy: + python-dev
messages: + msg218968
2014-05-21 15:02:56ronaldoussorensetmessages: + msg218878
2014-05-20 18:04:40serhiy.storchakasetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg218855
2014-05-20 17:31:24slo.sleuthsetstatus: open -> closed
resolution: fixed
messages: + msg218854
2014-05-20 16:58:39slo.sleuthsetmessages: + msg218850
2014-05-20 06:08:19serhiy.storchakasetfiles: - plistlib_read_refs.patch
2014-05-20 06:07:35serhiy.storchakasetfiles: + plistlib_read_refs_2.patch

messages: + msg218838
stage: test needed -> patch review
2014-05-20 05:32:41serhiy.storchakasetfiles: + plistlib_read_refs.patch
messages: + msg218835

keywords: + patch
type: behavior
stage: test needed
2014-05-20 05:05:02serhiy.storchakasetfiles: - History.plist.7z
2014-05-20 05:03:22serhiy.storchakasetfiles: + History.plist.7z
2014-05-20 04:41:43slo.sleuthsetfiles: + History.plist

messages: + msg218834
2014-05-20 00:51:46ned.deilysetnosy: + ned.deily
messages: + msg218827
2014-05-20 00:45:50ned.deilysetfiles: - History.plist.7z
2014-05-20 00:32:48ned.deilysetnosy: + ronaldoussoren, serhiy.storchaka, - ned.deily

type: crash -> (no value)
components: + Library (Lib)
versions: + Python 3.5
2014-05-20 00:09:57slo.sleuthsetfiles: + History.plist.7z

messages: + msg218825
2014-05-19 23:18:52ned.deilysetnosy: + ned.deily
messages: + msg218824
2014-05-19 22:08:52slo.sleuthcreate