Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MemoryError in test_strptime #69217

Closed
serhiy-storchaka opened this issue Sep 8, 2015 · 20 comments
Closed

MemoryError in test_strptime #69217

serhiy-storchaka opened this issue Sep 8, 2015 · 20 comments
Assignees
Labels
extension-modules C modules in the Modules dir release-blocker type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@serhiy-storchaka
Copy link
Member

BPO 25029
Nosy @abalkin, @vstinner, @larryhastings, @serhiy-storchaka, @eryksun, @zooba
Files
  • 25029_1.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/zooba'
    closed_at = <Date 2015-09-10.20:01:59.642>
    created_at = <Date 2015-09-08.10:30:13.592>
    labels = ['extension-modules', 'type-crash', 'release-blocker']
    title = 'MemoryError in test_strptime'
    updated_at = <Date 2015-09-10.20:01:59.640>
    user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2015-09-10.20:01:59.640>
    actor = 'steve.dower'
    assignee = 'steve.dower'
    closed = True
    closed_date = <Date 2015-09-10.20:01:59.642>
    closer = 'steve.dower'
    components = ['Extension Modules']
    creation = <Date 2015-09-08.10:30:13.592>
    creator = 'serhiy.storchaka'
    dependencies = []
    files = ['40407']
    hgrepos = []
    issue_num = 25029
    keywords = ['patch']
    message_count = 20.0
    messages = ['250177', '250190', '250192', '250197', '250199', '250201', '250205', '250206', '250207', '250212', '250215', '250219', '250225', '250266', '250271', '250276', '250280', '250312', '250404', '250416']
    nosy_count = 8.0
    nosy_names = ['belopolsky', 'vstinner', 'larry', 'python-dev', 'serhiy.storchaka', 'eryksun', 'steve.dower', 'David.Edelsohn']
    pr_nums = []
    priority = 'release blocker'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'crash'
    url = 'https://bugs.python.org/issue25029'
    versions = ['Python 3.5', 'Python 3.6']

    @serhiy-storchaka
    Copy link
    Member Author

    All builds on edelsohn-fedora-ppc64 are red starting from build 55 for 3.x and build 41 for 3.5. 3.4 is green.
    http://buildbot.python.org/all/builders/PPC64%20Fedora%203.x/builds/55/
    http://buildbot.python.org/all/builders/PPC64%20Fedora%203.5/builds/41/

    ======================================================================
    ERROR: test_TimeRE_recreation (test.test_strptime.CacheTests)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/home/shager/cpython-buildarea/3.5.edelsohn-fedora-ppc64/build/Lib/test/test_strptime.py", line 565, in test_TimeRE_recreation
        _strptime._strptime_time('10', '%d')
      File "/home/shager/cpython-buildarea/3.5.edelsohn-fedora-ppc64/build/Lib/_strptime.py", line 494, in _strptime_time
        tt = _strptime(data_string, format)[0]
      File "/home/shager/cpython-buildarea/3.5.edelsohn-fedora-ppc64/build/Lib/_strptime.py", line 312, in _strptime
        _TimeRE_cache = TimeRE()
      File "/home/shager/cpython-buildarea/3.5.edelsohn-fedora-ppc64/build/Lib/_strptime.py", line 190, in __init__
        self.locale_time = LocaleTime()
      File "/home/shager/cpython-buildarea/3.5.edelsohn-fedora-ppc64/build/Lib/_strptime.py", line 75, in __init__
        self.__calc_am_pm()
      File "/home/shager/cpython-buildarea/3.5.edelsohn-fedora-ppc64/build/Lib/_strptime.py", line 114, in __calc_am_pm
        am_pm.append(time.strftime("%p", time_tuple).lower())
    MemoryError

    @serhiy-storchaka serhiy-storchaka added extension-modules C modules in the Modules dir type-crash A hard crash of the interpreter, possibly with a core dump labels Sep 8, 2015
    @larryhastings
    Copy link
    Contributor

    Any clue here? Is this unaligned access?

    @DavidEdelsohn
    Copy link
    Mannequin

    DavidEdelsohn mannequin commented Sep 8, 2015

    PPC64 is not a strict alignment system. The system is running a non-recent release of Fedora, so it could be a bad interaction with libc.

    @vstinner
    Copy link
    Member

    vstinner commented Sep 8, 2015

    I ran test_strptime & test_os with tracemalloc: test_os memory peak is higher than test_strptime memory peak (I ran the two tests independently). So the bug cannot be reproduce on my Fedora 22 (x86_64).

    @vstinner
    Copy link
    Member

    vstinner commented Sep 8, 2015

    Oh wait, the VmPeak is *much* high when running test_strptime:

    • test_os VmPeak: 380 992 kB
    • test_strptime VmPeak: 8 608 972 kB (8 GB!?)

    1 GB should be enough for everybody:

    $ bash -c 'ulimit -v 1000000; ./python -m test -v test_strptime'
    ..

    ======================================================================
    ERROR: test_TimeRE_recreation (test.test_strptime.CacheTests)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      ...
      File "/home/haypo/prog/python/default/Lib/_strptime.py", line 75, in __init__
        self.__calc_am_pm()
      File "/home/haypo/prog/python/default/Lib/_strptime.py", line 114, in __calc_am_pm
        am_pm.append(time.strftime("%p", time_tuple).lower())
    MemoryError

    @vstinner
    Copy link
    Member

    vstinner commented Sep 8, 2015

    It's a regression introduced by c31dad22c80d (Issue bpo-24917).

    @abalkin
    Copy link
    Member

    abalkin commented Sep 8, 2015

    I would blame this change: <https://hg.python.org/cpython/rev/c31dad22c80d/#l3.41\>. The rest should not have any effect on Linux.

    @vstinner
    Copy link
    Member

    vstinner commented Sep 8, 2015

    Script to reproduce the issue:
    ---

    import time
    import locale
    import pprint
    
    time_tuple = time.struct_time((1999,3,17,1,44,55,2,76,0))
    
    p1 = time.strftime("%p", time_tuple)
    print("current LC_TIME", repr(p1))
    
    locale.setlocale(locale.LC_TIME, ('de_DE', 'UTF8'))
    p2 = time.strftime("%p", time_tuple)
    print("de_DE (UTF8)", repr(p2))

    Output:
    ---

    $ python3.4 bug.py
    current LC_TIME 'AM'
    de_DE (UTF8) ''

    The problem is that strftime()/wcsftime() *can* return 0, it's not an error. Whereas c31dad22c80d considers that if buflen is 0 but fmtlen is smaller than 5, we must retry with a larger buffer.

    @vstinner
    Copy link
    Member

    vstinner commented Sep 8, 2015

    Again, this issue remembers me the idea of rewriting strftime() in Python. We already have _strptime.py.

    @zooba
    Copy link
    Member

    zooba commented Sep 8, 2015

    Sorry Larry.

    I'll fix it.

    @zooba zooba self-assigned this Sep 8, 2015
    @eryksun
    Copy link
    Contributor

    eryksun commented Sep 8, 2015

    Steve, it seems to me that for MSVC the EINVAL test should come first:
       
        _Py_BEGIN_SUPPRESS_IPH
        olderr = errno;
        errno = 0;
        buflen = format_time(outbuf, i, fmt, &buf);
        err = errno;
        errno = olderr;
        _Py_END_SUPPRESS_IPH
        
        if (buflen == 0 && err == EINVAL) {
            PyErr_SetString(PyExc_ValueError, "Invalid format string");
            break;
        }

    Then the old test could be restored, i.e. (buflen > 0 || i >= 256 * fmtlen).

    @zooba
    Copy link
    Member

    zooba commented Sep 8, 2015

    @eryksun - that's exactly what I've just done :)

    Unfortunately, I don't have a _locale module, so I can't do Victor's test. Attached my patch in case Victor is around to test it.

    @vstinner
    Copy link
    Member

    vstinner commented Sep 8, 2015

    My test runs the following command on Linux (Fedora 22):

    make && bash -c 'ulimit -v 1000000; ./python -u -m test -v test_strptime'

    • current default branch (rev 3c0c153d6b02): MemoryError
    • with 25029_1.patch: the test pass

    @zooba
    Copy link
    Member

    zooba commented Sep 8, 2015

    Thanks. I'll submit a PR for 3.5.0 later tonight - can't seem to clone Larry's repo successfully at work for some reason.

    @zooba
    Copy link
    Member

    zooba commented Sep 9, 2015

    @larryhastings
    Copy link
    Contributor

    Does this need a Misc/NEWS entry?

    @zooba
    Copy link
    Member

    zooba commented Sep 9, 2015

    Probably should, since the fix that caused it was in for rc3. There's no section for 3.5.0 final yet though (that's my excuse, anyway :) )

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 9, 2015

    New changeset bd7aed300a1b by Steve Dower in branch '3.5':
    Issue bpo-25029: MemoryError in test_strptime
    https://hg.python.org/cpython/rev/bd7aed300a1b

    New changeset 706b9eaba734 by Steve Dower in branch '3.5':
    Merge fix for bpo-25029
    https://hg.python.org/cpython/rev/706b9eaba734

    New changeset cceaccb6ec5a by Steve Dower in branch '3.5':
    Adds Mics/NEWS entry for issue bpo-25029.
    https://hg.python.org/cpython/rev/cceaccb6ec5a

    @vstinner
    Copy link
    Member

    The test doesn't fail anymore on buildbots. I ran manually test_strptime with a memory limit of 1 GB on Python 3.5 and 3.6: the test pass.

    Can we close the issue?

    @zooba
    Copy link
    Member

    zooba commented Sep 10, 2015

    Done.

    @zooba zooba closed this as completed Sep 10, 2015
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    extension-modules C modules in the Modules dir release-blocker type-crash A hard crash of the interpreter, possibly with a core dump
    Projects
    None yet
    Development

    No branches or pull requests

    6 participants