classification
Title: Source files with date modifed in 2106 cause OverflowError
Type: crash Stage: resolved
Components: Interpreter Core Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: pitrou Nosy List: Arfrever, Guy.Kisel, barry, belopolsky, brett.cannon, haypo, ncoghlan, pitrou, python-dev
Priority: normal Keywords: patch

Created on 2011-02-17 20:20 by Guy.Kisel, last changed 2012-01-24 16:57 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
imptimestampoverflow.patch pitrou, 2012-01-24 09:19
Messages (9)
msg128753 - (view) Author: Guy Kisel (Guy.Kisel) Date: 2011-02-17 20:20
Tested in Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit (Intel)] on win32

Exception thrown: OverflowError: modification time overflows a 4 byte field

Steps to reproduce:
1. Set system time to 2/8/2106 or later and modify a .py file (or use a utility to change the date modified directly).
2. Try to run or import the .py file.

This date is 2^32 seconds after the Unix epoch.
msg128758 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-02-17 21:38
The problem occurs on import (import bla reads bla.py), when Python tries to create bla.pyc. The problem is that Python stores the timestamp as 4 bytes in .pyc files, whereas time_t is 64 bits on Windows (at least on Windows XP with Visual Studio).

To support bigger timestamps, we have to change the file format of .pyc files. It cannot be done in Python 2.7, I propose to do it in Python 3.3

See also #5537 and #4379: other issues with 64 bits => 32 bits timestamps.
msg128761 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-02-17 22:08
> To support bigger timestamps, we have to change the file format 
> of .pyc files.

write_compiled_module(), check_compiled_module() and other functions use the marshal module to read/write binary file, but marshal has no function for int64_t, only for long (which may be 32 bits, especially on Windows).

I don't know if Python has a builtin 64 bits integer type. There is PY_LONG_LONG, but this type is optional. A possible solution is to always store timestamp as 64 bits signed integer, but reject timestamp > 2^32 (as currently) if we don't have 64 bits integer type (like PY_LONG_LONG). Something like:

#ifdef PY_LONG_LONG
   write_timestamp64(t);
#else
   if (t << 32) { error; }
   write_timestamp32(t);
   write_long32(0); /* emulate 64 bits in big endian */
#endif
msg131594 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-03-21 01:17
By the way, a simpler fix would be just to not fail if the .pyc file cannot be created (but emit a warning / write an error to stderr).
msg151875 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-01-24 01:35
Fixing this is much easier than Victor's suggestion, we just have to ignore the higher bits of the timestamp (that is, store it modulo 2**32). This is enough for the purposes of testing the freshness of a pyc file.

And by avoiding modifying the pyc format, we can also ship the fix in bugfix releases.
msg151887 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-01-24 09:19
Here is a patch for 3.2. importlib doesn't have the issue, so I just added a test.
msg151914 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-01-24 16:10
LGTM
msg151915 - (view) Author: Roundup Robot (python-dev) Date: 2012-01-24 16:54
New changeset a2f3d6986bfa by Antoine Pitrou in branch '3.2':
Issue #11235: Fix OverflowError when trying to import a source file whose modification time doesn't fit in a 32-bit timestamp.
http://hg.python.org/cpython/rev/a2f3d6986bfa

New changeset cb13d8cff050 by Antoine Pitrou in branch 'default':
Issue #11235: Fix OverflowError when trying to import a source file whose modification time doesn't fit in a 32-bit timestamp.
http://hg.python.org/cpython/rev/cb13d8cff050

New changeset 0499eed74126 by Antoine Pitrou in branch '2.7':
Issue #11235: Fix OverflowError when trying to import a source file whose modification time doesn't fit in a 32-bit timestamp.
http://hg.python.org/cpython/rev/0499eed74126
msg151916 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-01-24 16:57
Committed, thanks.
History
Date User Action Args
2012-01-24 16:57:31pitrousetstatus: open -> closed
resolution: fixed
messages: + msg151916

stage: commit review -> resolved
2012-01-24 16:54:34python-devsetnosy: + python-dev
messages: + msg151915
2012-01-24 16:10:24brett.cannonsetassignee: pitrou
messages: + msg151914
stage: patch review -> commit review
2012-01-24 09:19:41pitrousetfiles: + imptimestampoverflow.patch
keywords: + patch
2012-01-24 09:19:25pitrousetmessages: + msg151887
stage: patch review
2012-01-24 01:35:53pitrousetversions: + Python 2.7, Python 3.2
nosy: + pitrou, brett.cannon

messages: + msg151875

components: + Interpreter Core, - None
2011-03-21 01:17:25hayposetnosy: barry, ncoghlan, belopolsky, haypo, Arfrever, Guy.Kisel
messages: + msg131594
2011-02-17 22:08:51hayposetnosy: barry, ncoghlan, belopolsky, haypo, Arfrever, Guy.Kisel
messages: + msg128761
2011-02-17 21:38:23hayposetnosy: + belopolsky, haypo

messages: + msg128758
versions: + Python 3.3, - Python 2.7
2011-02-17 20:50:49Arfreversetnosy: + Arfrever
2011-02-17 20:32:40pitrousetnosy: + barry, ncoghlan
2011-02-17 20:20:58Guy.Kiselcreate