Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

datetime.fromtimestamp() doesn't check min/max year anymore: regression of Python 3.6 #73286

Closed
JordonPhillips mannequin opened this issue Dec 29, 2016 · 18 comments
Closed
Labels
3.7 (EOL) end of life stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@JordonPhillips
Copy link
Mannequin

JordonPhillips mannequin commented Dec 29, 2016

BPO 29100
Nosy @malemburg, @abalkin, @vstinner, @dstufft, @eli-collins
PRs
  • [Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
  • Files
  • test.py
  • datetime_check_args.patch
  • unittest.patch
  • timestamp_limits.patch
  • timestamp_limits-2.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2017-02-10.09:36:06.392>
    created_at = <Date 2016-12-29.00:50:47.741>
    labels = ['3.7', 'type-bug', 'library']
    title = "datetime.fromtimestamp() doesn't check min/max year anymore: regression of Python 3.6"
    updated_at = <Date 2017-03-31.16:36:38.956>
    user = 'https://bugs.python.org/JordonPhillips'

    bugs.python.org fields:

    activity = <Date 2017-03-31.16:36:38.956>
    actor = 'dstufft'
    assignee = 'none'
    closed = True
    closed_date = <Date 2017-02-10.09:36:06.392>
    closer = 'vstinner'
    components = ['Library (Lib)']
    creation = <Date 2016-12-29.00:50:47.741>
    creator = 'Jordon Phillips'
    dependencies = []
    files = ['46068', '46124', '46131', '46142', '46437']
    hgrepos = []
    issue_num = 29100
    keywords = ['patch', '3.6regression']
    message_count = 18.0
    messages = ['284218', '284563', '284600', '284601', '284650', '286361', '286362', '286376', '287494', '287495', '287500', '287501', '287507', '287513', '287519', '287523', '287524', '290708']
    nosy_count = 7.0
    nosy_names = ['lemburg', 'belopolsky', 'vstinner', 'python-dev', 'dstufft', 'Eli Collins', 'Jordon Phillips']
    pr_nums = ['552']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue29100'
    versions = ['Python 3.6', 'Python 3.7']

    @JordonPhillips
    Copy link
    Mannequin Author

    JordonPhillips mannequin commented Dec 29, 2016

    In Python 3.6.0 if you give datetime.fromtimestamp a very bad value you either get a core dump or an OverflowError. The core dump occurs when no tzinfo is provided, the OverflowError occurs when a tzinfo is provided (such as tzlocal from dateutil). Attached is a minimal script to reproduce the error. Note that this behavior only occurs on certain systems. It does not happen on OSX 10.11.6, but it does happen on Ubuntu 12.04. I imagine it happens on other systems as well, but I haven't tested beyond those two.

    Here are the specific errors I get on Ubuntu 12.04. When no tzinfo is provided:

    python: /tmp/python-build.20161228223921.28011/Python-3.6.0/Modules/_datetimemodule.c:251: days_before_year: Assertion `year >= 1' failed.
    Aborted (core dumped)

    When a tzinfo is provided:

    Traceback (most recent call last):
      File "test1.py", line 11, in <module>
        datetime.fromtimestamp(bad_st_mtime, local_time_zone)
      File "~/.pyenv/versions/venv36d/lib/python3.6/site-packages/dateutil/tz/_common.py", line 210, in fromutc
        dt_wall = self._fromutc(dt)
      File "~/.pyenv/versions/venv36d/lib/python3.6/site-packages/dateutil/tz/_common.py", line 195, in _fromutc
        return dt + dtdst
    OverflowError: date value out of range

    I imagine this is related to the fold changes.

    @JordonPhillips JordonPhillips mannequin added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Dec 29, 2016
    @vstinner
    Copy link
    Member

    vstinner commented Jan 3, 2017

    Attached datetime_check_args.patch fixes the issue: move argument checks into time(), date() and datetime() constructors of the _datetime module.

    The patch requires the fix of the issue bpo-29140 (time_hash bug), otherwise unit tests fail.

    This issue is a regression introduced the implementation of the PEP-495. In Python 3.5, datetime_from_timet_and_us() calls datetime constructor which checks argument. In Python 3.6, datetime_from_timet_and_us() calls directly new_datetime_ex2() which doesn't check arguments anymore.

    datetime_check_args.patch makes the _datetime module generally safer, not only for the report bug, because new_datetime_ex2() can be called by different ways, for example from the C API: "PyDateTime_CAPI".

    The bug always shows a major lack of unit test in datetime.datetime.fromtimestamp() on dates out of the 0001-01-01 .. 9999-12-31 range. In Python 3.6, datetime.fromtimestamp() can create invalid datetime object greater than datetime.max!

    $ python3.6 -c 'import datetime; print(datetime.datetime.fromtimestamp(2**38))'
    10680-07-14 08:09:04

    TODO: add unit tests.

    @vstinner vstinner added the 3.7 (EOL) end of life label Jan 3, 2017
    @vstinner vstinner changed the title Core dump / OverflowError for datetime.fromtimestamp with overly large timestamp in Ubuntu 12.04 datetime.fromtimestamp() does crash for very small or very big timestamp Jan 3, 2017
    @vstinner vstinner changed the title datetime.fromtimestamp() does crash for very small or very big timestamp datetime.fromtimestamp() doesn't check min/max year anymore: regression of Python 3.6 Jan 3, 2017
    @vstinner
    Copy link
    Member

    vstinner commented Jan 3, 2017

    Oh, datetime.datetime.timestamp() also has a bug for values close to 0001-01-01 (year 1), at least in my timezone (CET).

    $ ./python -c 'import datetime; datetime.datetime.min.timestamp()'
    python: /home/haypo/prog/python/default/Modules/_datetimemodule.c:251: days_before_year: Assertion `year >= 1' failed.
    Aborted (core dumped)

    datetime_check_args.patch doesn't fix this issue. A solution is to check that year>=1 in utc_to_seconds():

    + if (year < MINYEAR || year > MAXYEAR) {
    + PyErr_SetString(PyExc_ValueError,
    + "year is out of range");
    + return -1;
    + }

    Maybe it would solve the following local() comment:

    /* XXX: add bounds checking */

    @vstinner
    Copy link
    Member

    vstinner commented Jan 3, 2017

    unitttest.patch: my attempt to write an unit test, but I was bitten by the datetime.timestamp() bug.

    @vstinner
    Copy link
    Member

    vstinner commented Jan 4, 2017

    timestamp_limits.patch: Add many checks in the _datetime module to avoid overflows or creating a object out of the range. The patch adds unit tests.

    The unit test checks dates around year 1 and year 9999. I expect failures on platforms which only work well on year in the range 1970..2038. My plan is to wait for buildbot reports to check on which platforms the test fails and with which exception.

    Since the patch is likely to break buildbots, I will apply first to the default branch, fix it, and then backport to Python 3.6.

    @vstinner
    Copy link
    Member

    The issue bpo-29346 has been marked as a duplicate of this one.

    @vstinner
    Copy link
    Member

    timestamp_limits.patch is waiting for your review! This issue is a major regression of Python 3.6. I'm going to push the patch in one week if I don't get any review.

    @vstinner
    Copy link
    Member

    timestamp_limits-2.patch: catch also OverflowError in unit tests, and uses a context manager rather than self.assertRaises(..., func, ...) syntax.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Feb 10, 2017

    New changeset 383c0238b5b0 by Victor Stinner in branch '3.6':
    Fix datetime.fromtimestamp(): check bounds
    https://hg.python.org/cpython/rev/383c0238b5b0

    @vstinner
    Copy link
    Member

    I pushed timestamp_limits-2.patch. Thanks for the bug report Jordon Phillips, thanks for the review Serhiy Storshaka.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Feb 10, 2017

    New changeset 5b804b2fb0eaa2bacb4afcfe4cfa85b31475e87f by Victor Stinner in branch '3.6':
    Fix datetime.fromtimestamp(): check bounds
    5b804b2

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Feb 10, 2017

    New changeset 5b804b2fb0eaa2bacb4afcfe4cfa85b31475e87f by Victor Stinner in branch 'master':
    Fix datetime.fromtimestamp(): check bounds
    5b804b2

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Feb 10, 2017

    New changeset 1c6b87b07586 by Victor Stinner in branch '3.6':
    Fix test_datetime on system with 32-bit time_t
    https://hg.python.org/cpython/rev/1c6b87b07586

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Feb 10, 2017

    New changeset a88f9614c672c3369cf03cdf51d012a42ae5665f by Victor Stinner in branch 'master':
    Fix test_datetime on system with 32-bit time_t
    a88f961

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Feb 10, 2017

    New changeset 1555e7776321 by Victor Stinner in branch '3.6':
    Fix test_datetime on Windows
    https://hg.python.org/cpython/rev/1555e7776321

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Feb 10, 2017

    New changeset be363764cad68ad7b608ceed837cc01c2c3b4efc by Victor Stinner in branch '3.6':
    Fix test_datetime on Windows
    be36376

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Feb 10, 2017

    New changeset be363764cad68ad7b608ceed837cc01c2c3b4efc by Victor Stinner in branch 'master':
    Fix test_datetime on Windows
    be36376

    @vstinner
    Copy link
    Member

    m-parry reported the issue bpo-29921 "datetime validation is stricter in 3.6.1 than previous versions". He is right, the C API of Python 3.6.1 is now stricter than Python 2.7 and 3.5. The C API doesn't allow anymore to create datetime objects outside the [datetime.datetime.min; datetime.datetime.max] range (what I would call "invalid" datetime objects). I closed the bug as NOT A BUG since I consider that it's a deliberate design choice and not a regression.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant