classification
Title: LWPCookieJar cannot handle cookies with expirations of 2038 or greater on 32-bit platforms
Type: behavior Stage:
Components: Library (Lib) Versions: Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Alex Quinn, belopolsky, haypo, hollec, jjlee, python-dev
Priority: normal Keywords: patch

Created on 2009-03-22 17:50 by hollec, last changed 2013-08-19 07:44 by Alex Quinn. This issue is now closed.

Files
File name Uploaded Description Edit
cookiejar_datetime.patch haypo, 2011-02-18 15:31
monkey_patch_cookielib_time2isoz.py Alex Quinn, 2013-08-19 07:39
Messages (13)
msg83979 - (view) Author: Chris Hollenbeck (hollec) Date: 2009-03-22 17:50
The LWPCookieJar can be saved on 64-bit Ubuntu, but not on 32-bit Ubuntu
when the expiration year is greater than 2038.  This has not been tested
on any other Intel-compatible Linux platform, though it appears related
to the Year 2038 bug.  The MozillaCookieJar does not have a problem
saving on either architecture.

A sample crash is shown below:

  File "/home/user/xblstatus/LiveConnect.py", line 189, in connect
    self.cookiejar.save(self.cookieFile)
  File "/usr/lib/python2.5/_LWPCookieJar.py", line 89, in save
    f.write(self.as_lwp_str(ignore_discard, ignore_expires))
  File "/usr/lib/python2.5/_LWPCookieJar.py", line 75, in as_lwp_str
    r.append("Set-Cookie3: %s" % lwp_cookie_str(cookie))
  File "/usr/lib/python2.5/_LWPCookieJar.py", line 35, in lwp_cookie_str
    time2isoz(float(cookie.expires))))
  File "/usr/lib/python2.5/cookielib.py", line 98, in time2isoz
    year, mon, mday, hour, min, sec = time.gmtime(t)[:6]
ValueError: timestamp out of range for platform time_t

---

The cookie jar and urllib2 integration was done with:

self.cookiejar = cookielib.LWPCookieJar()

self.opener =
urllib2.build_opener(urllib2.HTTPCookieProcessor(self.cookiejar))

urllib2.install_opener(self.opener)

---

The code used to save the cookie after accessing the web page was:

self.cookiejar.save(self.cookieFile)

The cookieFile variable is simply the default location of the cookie
file for saving in the program.
msg105416 - (view) Author: John J Lee (jjlee) Date: 2010-05-09 19:57
Shouldn't module time be changed to use a cross-platform implementation that uses a 64 bit time_t-like type?  Apparently Perl 6 has made the equivalent change.

Admittedly there seems to be no sign of that actually happening.
msg128765 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-02-17 22:26
> Shouldn't module time be changed to use a cross-platform implementation
> that uses a 64 bit time_t-like type? Apparently Perl 6 has made the
> equivalent change.

The error occurs on time.gmtime(t): even if we use 64 bits time_t type, we have to downcast it to system time_t later, because we would like to call gmtime() function at the end.

To workaround gmtime() limitation: we can simply use datetime instead. Attached patch replaces gmtime() by datetime.utcfromtimestamp(), and use its .strftime() method which has no such limitation.
msg128781 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-02-18 13:31
Oh, my patch is incomplete: time2netscape() has the same issue.
msg128788 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-02-18 15:29
Victor,

I don't see your patch.  Did you remove it?
msg128789 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-02-18 15:31
No, I forgot to upload it...
msg128791 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-02-18 15:43
While it is unlikely that a purely numeric format such as "%Y-%m-%d
%H:%M:%S" will be locale dependent, it is possible that some pre-C99
systems would format dates using exotic digits is some locales.  Given
that format is so simple, I would just use explicit formatting of dt
components instead of datetime.strftime.  Doing so will also eliminate
a call to system strftime which is known to be quirky on popular
platforms even in C locale.
msg128792 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-02-18 16:04
If datetime.strftime() is not reliable, we should maybe fix it instead of using a workaround?
msg128793 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-02-18 16:12
On Fri, Feb 18, 2011 at 11:04 AM, STINNER Victor <report@bugs.python.org> wrote:
..
> If datetime.strftime() is not reliable, we should maybe fix it instead of using a workaround?

The real fix would be to rewrite strftime so that it does not call
system strftime.  See issue3173.
msg131603 - (view) Author: Roundup Robot (python-dev) Date: 2011-03-21 01:53
New changeset b15f60f9e256 by Victor Stinner in branch '3.1':
Issue #5537: Fix time2isoz() and time2netscape() functions of httplib.cookiejar
http://hg.python.org/cpython/rev/b15f60f9e256
msg131604 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-03-21 02:04
> While it is unlikely that a purely numeric format such as "%Y-%m-%d
> %H:%M:%S" will be locale dependent, it is possible that some pre-C99
> systems would format dates using exotic digits is some locales.

Ok, I rewrote my patch to avoid strftime(). It should now be fixed.

FYI datetime.fromtimestamp() converts the timestamp to a double, which has a precision of 53 bits (no precision loss for year < 285,422,890 and so it's enough for year 2038).
msg195609 - (view) Author: Alex Quinn (Alex Quinn) Date: 2013-08-19 07:39
This bug still exists in Python 2.7.3 32-bit on Linux.

I wonder if this might be because the patch (posted 2011-02-18) used utcfromtimestamp().  datetime.datetime.utcfromtimestamp(2**32) will fail on 32-bit systems.

The bug does NOT exist in Python 2.7.3 32-bit on Windows (64-bit OS).

==========================================
32-BIT PYTHON ON 32-BIT LINUX
------------------------------------------
$ python -c "import sys, cookielib; print sys.version; print cookielib.time2isoz(2322923767)"
2.7.3 (default, Apr 10 2013, 05:46:21) 
[GCC 4.6.3]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/cookielib.py", line 99, in time2isoz
    year, mon, mday, hour, min, sec = time.gmtime(t)[:6]
ValueError: timestamp out of range for platform time_t
==========================================


==========================================
64-BIT PYTHON ON 64-BIT LINUX
------------------------------------------
$ python -c "import sys, cookielib; print sys.version; print cookielib.time2isoz(2322923767)"
2.7.3 (default, Aug  3 2012, 17:21:07)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)]
2043-08-11 16:36:07Z
==========================================


==========================================
32-BIT PYTHON ON 64-BIT WINDOWS
------------------------------------------
C:\>python -c "import sys, cookielib; print sys.version; print cookielib.time2isoz(2322923767)"
2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)]
2043-08-11 16:36:07Z
==========================================
msg195610 - (view) Author: Alex Quinn (Alex Quinn) Date: 2013-08-19 07:44
For those who are affected by this bug, here's a snippet to monkey-patch cookielib on any Python 2.4 to 2.7.

A more complete version of this was attached to my message a moment ago.

==========================================
import cookielib
try:
    cookielib.time2isoz(2**32)
except ValueError:
    from datetime import datetime, timedelta
    def time2isoz(t=None):
        if t is None:
            dt = datetime.now()
        else:
            dt = datetime.utcfromtimestamp(0) + timedelta(seconds=int(t))
        return "%04d-%02d-%02d %02d:%02d:%02dZ"%\
                (dt.year, dt.month, dt.day, dt.hour, dt.minute, dt.second)
    cookielib.time2isoz = time2isoz
==========================================
History
Date User Action Args
2013-08-19 07:44:41Alex Quinnsetmessages: + msg195610
2013-08-19 07:40:00Alex Quinnsetfiles: + monkey_patch_cookielib_time2isoz.py
nosy: + Alex Quinn
messages: + msg195609

2011-03-21 02:04:29hayposetstatus: open -> closed
nosy: jjlee, belopolsky, haypo, hollec, python-dev
resolution: fixed
2011-03-21 02:04:20hayposetnosy: jjlee, belopolsky, haypo, hollec, python-dev
messages: + msg131604
2011-03-21 01:53:20python-devsetnosy: + python-dev
messages: + msg131603
2011-02-18 16:12:29belopolskysetnosy: jjlee, belopolsky, haypo, hollec
messages: + msg128793
2011-02-18 16:04:29hayposetnosy: jjlee, belopolsky, haypo, hollec
messages: + msg128792
2011-02-18 15:43:52belopolskysetnosy: jjlee, belopolsky, haypo, hollec
messages: + msg128791
2011-02-18 15:31:01hayposetfiles: + cookiejar_datetime.patch

messages: + msg128789
keywords: + patch
nosy: jjlee, belopolsky, haypo, hollec
2011-02-18 15:29:57belopolskysettype: behavior
messages: + msg128788
nosy: jjlee, belopolsky, haypo, hollec
2011-02-18 13:31:44hayposetnosy: jjlee, belopolsky, haypo, hollec
messages: + msg128781
2011-02-17 22:26:46hayposetnosy: + haypo, belopolsky
messages: + msg128765
2010-05-09 19:57:30jjleesetnosy: + jjlee
messages: + msg105416
2009-03-22 17:50:30holleccreate