classification
Title: Python implementation of datetime module is not being tested correctly.
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: belopolsky, musically_ut, ned.deily, serhiy.storchaka, vstinner
Priority: normal Keywords:

Created on 2017-06-30 23:47 by musically_ut, last changed 2017-09-08 22:44 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 2530 merged musically_ut, 2017-07-01 23:44
PR 2534 merged serhiy.storchaka, 2017-07-02 17:55
PR 2550 closed musically_ut, 2017-07-03 19:37
PR 2551 closed musically_ut, 2017-07-03 19:55
PR 2588 merged vstinner, 2017-07-05 12:44
PR 2775 merged vstinner, 2017-07-20 11:43
PR 2777 merged vstinner, 2017-07-20 13:59
PR 2781 merged vstinner, 2017-07-20 14:16
PR 2782 merged vstinner, 2017-07-20 15:16
PR 2783 merged musically_ut, 2017-07-20 16:15
PR 2788 merged vstinner, 2017-07-20 23:21
PR 2815 closed musically_ut, 2017-07-22 15:40
PR 2816 merged musically_ut, 2017-07-22 15:47
PR 3405 merged python-dev, 2017-09-07 00:01
Messages (48)
msg297455 - (view) Author: Utkarsh Upadhyay (musically_ut) * Date: 2017-06-30 23:47
While investigating http://bugs.python.org/issue30302 it was discovered that the Python implementation of the datetime module was not being tested correctly.

The datetimetester.py was supposed to execute tests twice, once for the _Fast implementation (i.e. C extension code in _datetimemodule.c) and once for the _Pure implementation (i.e the Python code).

The fix is contained in the following two commits:

 - https://github.com/python/cpython/pull/1493/commits/08e7548f56838fca43b488cefe51de4bdd600f3d
 - https://github.com/python/cpython/pull/1493/commits/94d5a4e4d33a1d14a2bb1be8fbff5e1e4cd2b7a6

The bug does not effect Python 2.7 as the C version of the datetime module had not been introduced back then. However, as fas as I can tell, the fix needs to be applied to all Python 3.x branches.
msg297470 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-01 04:45
Please provide a separate PR for this. After merging it with the master branch it should be backported to 3.6 and 3.5 (3.4 and 3.3 are for security only fixes). Add your name in Misc/ACKS. Changes in Misc/NEWS.d/ are not required.
msg297515 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-02 12:46
New changeset 98b6bc3bf72532b784a1c1fa76eaa6026a663e44 by Serhiy Storchaka (Utkarsh Upadhyay) in branch 'master':
bpo-30822: Fix testing of datetime module. (#2530)
https://github.com/python/cpython/commit/98b6bc3bf72532b784a1c1fa76eaa6026a663e44
msg297517 - (view) Author: Utkarsh Upadhyay (musically_ut) * Date: 2017-07-02 13:52
Are there any guides re: backporting commits?

Just to confirm verify that I'm going about it right; I’m planning on cherry-picking relevant commits back to branch 3.5, 3.6, resolving the merge conflicts, and then opening two separate PRs for them with `bpo-30822` in their titles.

Sounds about right?

~
ut
msg297518 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-07-02 16:12
Serhiy, it appears that the added tests are taking a long time to run and thus causing many buildbot failures, for example:

http://buildbot.python.org/all/builders/ARMv7%20Ubuntu%203.x/builds/1013
msg297519 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-02 16:17
Oh, test.datetimetester.ZoneInfoTest[Europe/Andorra]_Pure_Pure_Pure -- something looks wrong.
msg297520 - (view) Author: Utkarsh Upadhyay (musically_ut) * Date: 2017-07-02 17:11
I can reproduce the names with `_Pure_Pure_Pure` if I run the tests with `-utzdata`. Investigating.

~
ut
msg297522 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-02 18:02
The quick solution -- just deduplicate test classes.

But the code needs rewriting. Current code is tricky and fragile. It doesn't work with --list-cases (and I suppose it doesn't work well with other advanced unittest features). This would be not easy.
msg297523 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-02 18:07
> Are there any guides re: backporting commits?
>
> Just to confirm verify that I'm going about it right; I’m planning on cherry-picking relevant commits back to branch 3.5, 3.6, resolving the merge conflicts, and then opening two separate PRs for them with `bpo-30822` in their titles.

All right. See https://docs.python.org/devguide/committing.html#backporting-changes-to-an-older-version . You can cherry-pick manually or using cherry_picker.py which adds correct prefix to commit message and helps to create a pull request.
msg297524 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-02 19:36
New changeset 34b54873b51a1ebee2a3c57b7205537b4f33128d by Serhiy Storchaka in branch 'master':
bpo-30822: Deduplicate ZoneInfoTest classes in test_datetime. (#2534)
https://github.com/python/cpython/commit/34b54873b51a1ebee2a3c57b7205537b4f33128d
msg297525 - (view) Author: Utkarsh Upadhyay (musically_ut) * Date: 2017-07-02 19:37
So the problem is occurring because a single `Test` class is being instantiated with three different tests here in datetimetester.py:

            for name in ZoneInfo.zonenames():
                Test = type('ZoneInfoTest[%s]' % name, (ZoneInfoTest,), {})
                Test.zonename = name
                for method in dir(Test):
                    if method.startswith('test_'):
                        tests.append(Test(method))

here: https://github.com/python/cpython/blob/master/Lib/test/datetimetester.py#L4853

The `Test` class which is being dynamically created is being instantiated with the test methods: test_folds, test_gaps, test_system_transitions and is being appended to tests. This makes that class to appear thrice in `test_classes` in `test_datetime.py`:

        elif issubclass(cls, unittest.TestSuite):
            suit = cls()
            test_classes.extend(type(test) for test in suit)

here: https://github.com/python/cpython/blob/master/Lib/test/test_datetime.py#L34

Hence, the class gets `_Pure` and `_Fast` appended to its name thrice and gets executed thrice, making the tests take 3 times as long to run.

This is confirmed by changing the code to the following:

           for name in ZoneInfo.zonenames():
                Test = type('ZoneInfoTest[%s]' % name, (ZoneInfoTest,), {})
                Test.zonename = name
                tests.append(Test())
                # for method in dir(Test):
                #    if method.startswith('test_'):
                #        tests.append(Test(method))

However, I'm not sure what implications this has w.r.t. unittests and the advanced cases.

The other way to fix it is to create a set out of the classes, as suggested in PR #2534.

~
ut
msg297530 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-03 04:09
> This is confirmed by changing the code to the following:

Yes, this is other way to fix test_datetime. But this breaks running tests via datetimetester:

   ./python -m test -vuall datetimetester
msg297546 - (view) Author: Utkarsh Upadhyay (musically_ut) * Date: 2017-07-03 08:33
Hmm, I see; I did not know that.

I had just assumed that `./python -m test test_*` was the standard command for running tests.

However, if one does run tests using standalone `datetimetester` (which only imports the _Fast C extension module), then I'll have to rewrite some of the self.skipTest conditions such that they are more defensive, i.e.:

    if '_Pure' not in self.__class__.__name__:
         self.skipTest('...')

instead of:

    if '_Fast' in self.__class__.__name__:
         self.skipTest('...')


because some (one?) of the tests fail otherwise.

Shall I make that change?

Also, here is an alternate fix which is a tiny bit less murky than straight up de-duplication of test-classes: https://github.com/musically-ut/cpython/pull/1/files

I'm not sure whether this is any clearer or less fragile, though.

~
ut
msg297548 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-03 09:01
Good point!

But datetimetester uses the implementation depending on the availability of the _datetime extension module. If it is not available, the pure Python implementation is used, and other tests are failed.

I think it would be better to fix first the issues with test_datetime, since this is the common way of running tests, and try to restore running tests with datetimetester in separate issue. Just backport your PR 2530 to 3.5 and 3.6. You can include the change from PR 2534 or it can be backported separately.
msg297554 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-03 10:41
Regression:

http://buildbot.python.org/all/builders/PPC64%20Fedora%203.x/builds/1000/steps/test/logs/stdio

test_system_transitions (test.datetimetester.ZoneInfoTest[America/Juneau]_Pure_Pure_Pure) ... Timeout (0:15:00)!
Thread 0x00003fffa9aa5a20 (most recent call first):
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/datetime.py", line 454 in __new__
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/datetime.py", line 1870 in __sub__
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/datetime.py", line 1519 in local
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/datetime.py", line 1532 in _mktime
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/datetime.py", line 1550 in timestamp
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/test/datetimetester.py", line 4836 in test_system_transitions
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/unittest/case.py", line 615 in run
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/unittest/case.py", line 663 in __call__
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/unittest/suite.py", line 122 in run
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/unittest/suite.py", line 122 in run
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/unittest/runner.py", line 176 in run
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/test/support/__init__.py", line 1896 in _run_suite
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/test/support/__init__.py", line 1940 in run_unittest
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/test/test_datetime.py", line 53 in test_main
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/test/libregrtest/runtest.py", line 172 in runtest_inner
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/test/libregrtest/runtest.py", line 140 in runtest
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/test/libregrtest/main.py", line 290 in rerun_failed_tests
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/test/libregrtest/main.py", line 540 in _main
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/test/libregrtest/main.py", line 510 in main
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/test/libregrtest/main.py", line 585 in main
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/test/__main__.py", line 2 in <module>
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/runpy.py", line 85 in _run_code
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/runpy.py", line 193 in _run_module_as_main
make: *** [buildbottest] Error 1
program finished with exit code 2
elapsedTime=2290.631264
msg297555 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-03 10:42
http://buildbot.python.org/all/builders/ARMv7%20Ubuntu%203.x/builds/1013/steps/test/logs/stdio


test_gaps (test.datetimetester.ZoneInfoTest[Europe/Riga]_Pure_Pure_Pure) ... ok
test_system_transitions (test.datetimetester.ZoneInfoTest[Europe/Riga]_Pure_Pure_Pure) ... ok
test_folds (test.datetimetester.ZoneInfoTest[Europe/Riga]_Pure_Pure_Pure) ... ok
test_gaps (test.datetimetester.ZoneInfoTest[Europe/Riga]_Pure_Pure_Pure) ... ok
test_system_transitions (test.datetimetester.ZoneInfoTest[Europe/Riga]_Pure_Pure_Pure) ... ok
test_folds (test.datetimetester.ZoneInfoTest[Europe/Riga]_Pure_Pure_Pure) ... Timeout (0:15:00)!
Thread 0x400c9110 (most recent call first):
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/datetime.py", line 263 in _check_int_field
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/datetime.py", line 281 in _check_date_fields
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/datetime.py", line 1376 in __new__
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/datetime.py", line 1600 in replace
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/datetimetester.py", line 4776 in test_folds
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/case.py", line 615 in run
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/case.py", line 663 in __call__
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/suite.py", line 122 in run
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/suite.py", line 84 in __call__
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/suite.py", line 122 in run
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/suite.py", line 84 in __call__
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/runner.py", line 176 in run
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/support/__init__.py", line 1896 in _run_suite
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/support/__init__.py", line 1940 in run_unittest
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_datetime.py", line 53 in test_main
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/runtest.py", line 172 in runtest_inner
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/runtest.py", line 140 in runtest
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/main.py", line 290 in rerun_failed_tests
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/main.py", line 540 in _main
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/main.py", line 510 in main
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/main.py", line 585 in main
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/__main__.py", line 2 in <module>
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/runpy.py", line 85 in _run_code
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/runpy.py", line 193 in _run_module_as_main
make: *** [buildbottest] Error 1
program finished with exit code 2
elapsedTime=2225.001566
msg297557 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-03 10:50
Oh, I didn't read the issue. In fact, it seems like the commit 34b54873b51a1ebee2a3c57b7205537b4f33128d already repaired buildbots (x86 Tiger 3.x at least).

But... test_datetime became the *slowest* test of x86 Tiger 3.x:

10 slowest tests:
- test_datetime: 19 min 6 sec
- test_tools: 7 min 20 sec
- test_multiprocessing_spawn: 6 min 54 sec
- ...

Wait, 19 minutes to test datetime??? What's wrong with test_datetime?
msg297559 - (view) Author: Utkarsh Upadhyay (musically_ut) * Date: 2017-07-03 10:59
Previously, test_datetime was running only `_Fast` tests (i.e. testing the C module). Then (because of an erring commit from me), testing for `_Pure` implementation started but all tests per time-zone started running thrice instead of once. Hence, the timeouts while testing were triggered, breaking the buildbots. (Err ... sorry about that.)

However, the latest fix by Serihy now runs each test only once. The `_Pure` tests, nevertheless, are significantly slower than `_Fast` tests and, hence, the slowdown.

Running the tests take ~ 10 minutes on my system with -utzdata enabled; 19 minutes does sound like a bit much but I'm not aware of the specs of the buildbot machines.

~
ut
msg297562 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-03 11:12
> Running the tests take ~ 10 minutes on my system with -utzdata enabled; 19 minutes does sound like a bit much but I'm not aware of the specs of the buildbot machines.

I don't consider that 10 minutes to test the small datetime module is a reasonable duration. What does take such more time? Do we really need to test all timezones around the world?
msg297571 - (view) Author: Utkarsh Upadhyay (musically_ut) * Date: 2017-07-03 11:32
Hmm, I don't know if testing every zone is necessary. However, I would not be comfortable picking out the zones one ought to test, considering:

         if ('Riyadh8' in self.zonename or
            # From tzdata NEWS file:
            # The files solar87, solar88, and solar89 are no longer distributed.
            # They were a negative experiment - that is, a demonstration that
            # tz data can represent solar time only with some difficulty and error.
            # Their presence in the distribution caused confusion, as Riyadh
            # civil time was generally not solar time in those years.
self.zonename.startswith('right/')):

and:

> # Iran had a sub-minute UTC offset before 1946.

in https://github.com/python/cpython/blob/master/Lib/test/datetimetester.py#L4865

Perhaps @belopolsky can suggest something? (Already in Nosy List).

~
ut
msg297573 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-03 11:37
Let discuss too long time of testing datetime time zones in separate issue.
msg297612 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-03 20:07
I would prefer to make test_datetime faster in master before backporting.
msg297734 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-05 12:42
It's not just a matter of making test_datetime "faster". I see it as a regression, since now it fails randomly on slow buildbots.

Maybe we should revert the change until a solution is found.

http://buildbot.python.org/all/builders/ARMv7%20Ubuntu%203.x/builds/1025/steps/test/logs/stdio

running: test_datetime (802 sec)
running: test_datetime (832 sec)
running: test_datetime (862 sec)
running: test_datetime (892 sec)
0:15:02 load avg: 2.30 [406/406/1] test_datetime crashed (Exit code 1)
Timeout (0:15:00)!
Thread 0x400b4110 (most recent call first):
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/datetime.py", line 394 in __new__
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/datetime.py", line 1519 in local
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/datetime.py", line 1522 in _mktime
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/datetime.py", line 1550 in timestamp
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/datetimetester.py", line 4836 in test_system_transitions
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/case.py", line 615 in run
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/case.py", line 663 in __call__
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/suite.py", line 122 in run
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/suite.py", line 84 in __call__
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/suite.py", line 122 in run
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/suite.py", line 84 in __call__
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/runner.py", line 176 in run
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/support/__init__.py", line 1896 in _run_suite
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/support/__init__.py", line 1940 in run_unittest
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_datetime.py", line 54 in test_main
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/runtest.py", line 172 in runtest_inner
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/runtest.py", line 130 in runtest
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/runtest_mp.py", line 71 in run_tests_slave
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/main.py", line 519 in _main
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/main.py", line 512 in main
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/main.py", line 587 in main
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/regrtest.py", line 46 in _main
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/regrtest.py", line 50 in <module>
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/runpy.py", line 85 in _run_code
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/runpy.py", line 193 in _run_module_as_main
msg297736 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-05 13:13
Another solution -- disable "cpu" and "tzdata" resources on slow buildbots (see issue30417).
msg297743 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-05 13:44
New changeset 8207c17486baece8ed0ac42d9f8d69ecec4ba7e4 by Victor Stinner in branch 'master':
Revert "bpo-30822: Fix testing of datetime module." (#2588)
https://github.com/python/cpython/commit/8207c17486baece8ed0ac42d9f8d69ecec4ba7e4
msg297752 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-05 13:45
I reverted the change to repair buildbots and get more time to find a proper fix:
https://github.com/python/cpython/pull/2588#issuecomment-313092304
msg297753 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-05 13:47
> Another solution -- disable "cpu" and "tzdata" resources on slow buildbots (see issue30417).

I didn't read test_datetime. How test_datetime can spend 20 minutes to test timestamps? Does it spawn subprocesses? Why is it so slow?
msg297760 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2017-07-05 14:09
> Why is it so slow?

The tests enabled by "-utzdata" check UTC to local and back conversions at several points around *every* time transition in *every* timezone. On systems with a complete installation of IANA tzdata, this is a lot of test points.

These tests were supposed to be exhaustive and I did not expect them to be run by default.  that's why I introduced the -utzdata flag in the first place.
msg297762 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-05 14:28
> These tests were supposed to be exhaustive and I did not expect them to be run by default.  that's why I introduced the -utzdata flag in the first place.

Buildbots use "-u all" and so enables tzdata tests. Is it useful to
run all these tests on all branches for each Python commit?

Maybe we should disable these tests on all CI, and maybe setup a
buildbot which runs these tests?

Instead of being exhaustive, would it be possible to pick a few
significant tests?
msg297763 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-05 14:58
Two timezones (America/New_York and Asia/Tehran) are picked for testing independently from the -utzdata flag.
msg298032 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-10 08:39
I reverted the commit which fixes test_datetime to also test Lib/datetime.py, to repair buildbots. But since I reverted the change, nothing was done so datetime.py is still not tested :-(

> The tests enabled by "-utzdata" check UTC to local and back conversions at several points around *every* time transition in *every* timezone. On systems with a complete installation of IANA tzdata, this is a lot of test points.
>
> These tests were supposed to be exhaustive and I did not expect them to be run by default.  that's why I introduced the -utzdata flag in the first place.

Alexander: yeah, having an opt-in option to test all timezones makes sense. It's likely to trigger bugs in some corner cases. But the question is really having our CI.

Alexander, Serhiy: would you be ok to disable tzdata resource on all our CI (Travis CI, AppVeyor, all buildbots)?

*Maybe* we might enable tzdata on selected (fast) buildbots where test_datetime takes less than 20 minutes. But I would prefer to keep the option as an *opt-in*, rather than always running all tests on all CI.
msg298033 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-10 08:40
> Two timezones (America/New_York and Asia/Tehran) are picked for testing independently from the -utzdata flag.

By the way, if someone is aware of other interesting timezones, we may also test more timezones (without tzdata)?
msg298059 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-10 12:37
If testing with -u tzdata is such expensive, tzdata can be excluded from all resources. There is a precedence with the extralargefile resource (see test_zipfile64).

Actually this means that these test will never run until you request this specially.
msg298664 - (view) Author: Utkarsh Upadhyay (musically_ut) * Date: 2017-07-19 10:21
So it seems that running the experiments without `-utzdata` would be an acceptable fix (assuming that there are no other interesting time-zones worthy of explicit testing)?

Can I help in the resolution of this issue, since resolution of http://bugs.python.org/issue30302 is tangentially conditioned on it. (-:
msg298717 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-20 12:28
Serhiy Storchaka: "If testing with -u tzdata is such expensive, tzdata can be excluded from all resources."

I agree, I wrote a PR for that:
https://github.com/python/cpython/pull/2775

I like using "-u all" without having to remind that a specific resource (tzdata) is too slow to be enabled by default.

I suggest to backport this change to all branches.
msg298721 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-20 13:46
New changeset 5b392bbaeb9d9b1db961ecfc7315d8c8662c27f6 by Victor Stinner in branch 'master':
bpo-30822: Exclude tzdata from regrtest --all (#2775)
https://github.com/python/cpython/commit/5b392bbaeb9d9b1db961ecfc7315d8c8662c27f6
msg298725 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-20 15:08
New changeset 96ef06f3979023fd69ed8796d0b1d88885d3ae4d by Victor Stinner in branch '3.6':
bpo-30822: Exclude tzdata from regrtest --all (#2775) (#2777)
https://github.com/python/cpython/commit/96ef06f3979023fd69ed8796d0b1d88885d3ae4d
msg298726 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-20 15:08
New changeset 645e503ba58086c7f51eda4d025743f2afc05a2a by Victor Stinner in branch '3.5':
bpo-30822: Exclude tzdata from regrtest --all (#2775) (#2781)
https://github.com/python/cpython/commit/645e503ba58086c7f51eda4d025743f2afc05a2a
msg298729 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-20 15:31
(Oops, I gone too fast in my 3.5 backport. tzdata was only introduced in Python 3.6, for me it was something older. I wrote a new fix for 3.5, to only add extralargefile, but don't add tzdata.)
msg298732 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-20 16:08
New changeset bf3a1e973b49236e6f267cca135d36290568e6a4 by Victor Stinner in branch '3.5':
bpo-30822: regrtest: remove tzdata (#2782)
https://github.com/python/cpython/commit/bf3a1e973b49236e6f267cca135d36290568e6a4
msg298755 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-21 00:12
New changeset 80ebc438ed377f8c6491e8887425526c6787c0e7 by Victor Stinner in branch '2.7':
bpo-30822: regrtest: fix -u extralargefile (#2788)
https://github.com/python/cpython/commit/80ebc438ed377f8c6491e8887425526c6787c0e7
msg298756 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-21 00:14
New changeset 287c5594edc1ca08db64d1f4739cc36bfe75ae75 by Victor Stinner (Utkarsh Upadhyay) in branch 'master':
bpo-30822: Fix testing of datetime module. (#2530) (#2783)
https://github.com/python/cpython/commit/287c5594edc1ca08db64d1f4739cc36bfe75ae75
msg298786 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-21 09:13
Utkarsh comment on the PR:
> I'm afraid I don't really know how to read the waterfall chart at http://buildbot.python.org/all/waterfall.

Yeah, sorry about this awful view :-) We now have a mailing list getting notifications when a buildbot fails:
https://mail.python.org/mm3/mailman3/lists/buildbot-status.python.org/

We got two new failures since yesterday on Windows buildbots, but it's related to IDLE and so unrelated to your change. So a quick check says that test_datetime doesn't time out anymore!

Previously, the test failed on a few buildbots. I checked the test output to look if test_datetime became the new slowest test and ... no. For example, on PPC64 Fedora 3.x, test_datetime is not even listed in the "10 slowest tests", which means that it took 36 sec or less, whereas the test took longer than 15 minutes when tzdata was used.

IMHO you can already cook a first backport for 3.6, but I would prefer to wait another day just to make sure to everything is fine (to wait for new buildbot builds).
msg299221 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-26 11:49
New changeset c52cea49544621b612c7f17f45a0c2b8b61a6c67 by Victor Stinner (Utkarsh Upadhyay) in branch '3.6':
[3.6] bpo-30822: Fix testing of datetime module. (GH-2530) (GH-2783) (#2816)
https://github.com/python/cpython/commit/c52cea49544621b612c7f17f45a0c2b8b61a6c67
msg299222 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-26 11:52
Ok, I merged the change in the 3.6 branch.

I decided to not merge the 3.5 change, since it's now too late: 3.5 slowly enters a new security-only fixes:
https://github.com/python/cpython/pull/2815

Thanks Utkarsh Upadhyay for your work on datetime, not only this issue, but also repr(timedelta) (and maybe also timedelta constructor doc later? ;-)).
msg299223 - (view) Author: Utkarsh Upadhyay (musically_ut) * Date: 2017-07-26 11:58
Thanks Victor! \o/

Secretly, I'm just happy that my legacy will not be a commit which broke all (?) the build-bots and had to be reverted. :P

W.r.t. the docs; in retrospect, that'll probably have a larger impact on the end-users and is less likely to cause disagreement. I probably should have led with that. :)

~
ut
msg299224 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-26 12:04
> Secretly, I'm just happy that my legacy will not be a commit which broke all (?) the build-bots and had to be reverted. :P

Reverting is a new policy that we decided last June:
https://mail.python.org/pipermail/python-committers/2017-June/004588.html

Sorry that your commit was the first guilty, but honestly, it was a wise decision since it allowed to discuss how to handle tzdata without the pressure of broken buildbots. Nowadays, Python relies much more on CI for pull requests than before the migration to GitHub.

Well, read the thread if you want to know the full rationale.
msg301749 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-08 22:44
New changeset 3892799668dbf2b123a52780fd1d78f8880fdeb7 by Victor Stinner (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-30822: Deduplicate ZoneInfoTest classes in test_datetime. (GH-2534) (#3405)
https://github.com/python/cpython/commit/3892799668dbf2b123a52780fd1d78f8880fdeb7
History
Date User Action Args
2017-09-08 22:44:35vstinnersetmessages: + msg301749
2017-09-07 00:01:54python-devsetpull_requests: + pull_request3408
2017-07-26 12:04:59vstinnersetmessages: + msg299224
2017-07-26 11:58:20musically_utsetmessages: + msg299223
2017-07-26 11:52:55vstinnersetstatus: open -> closed
versions: - Python 3.5
messages: + msg299222

resolution: fixed
stage: backport needed -> resolved
2017-07-26 11:49:18vstinnersetmessages: + msg299221
2017-07-22 15:47:31musically_utsetpull_requests: + pull_request2868
2017-07-22 15:40:49musically_utsetpull_requests: + pull_request2867
2017-07-21 09:13:03vstinnersetmessages: + msg298786
2017-07-21 00:14:56vstinnersetmessages: + msg298756
2017-07-21 00:12:18vstinnersetmessages: + msg298755
2017-07-20 23:21:47vstinnersetpull_requests: + pull_request2838
2017-07-20 16:15:43musically_utsetpull_requests: + pull_request2834
2017-07-20 16:08:57vstinnersetmessages: + msg298732
2017-07-20 15:31:31vstinnersetmessages: + msg298729
2017-07-20 15:16:27vstinnersetpull_requests: + pull_request2833
2017-07-20 15:08:54vstinnersetmessages: + msg298726
2017-07-20 15:08:50vstinnersetmessages: + msg298725
2017-07-20 14:16:00vstinnersetpull_requests: + pull_request2832
2017-07-20 13:59:58vstinnersetpull_requests: + pull_request2831
2017-07-20 13:46:35vstinnersetmessages: + msg298721
2017-07-20 12:28:08vstinnersetmessages: + msg298717
2017-07-20 11:43:29vstinnersetpull_requests: + pull_request2829
2017-07-19 10:21:32musically_utsetmessages: + msg298664
2017-07-10 12:37:31serhiy.storchakasetmessages: + msg298059
2017-07-10 08:40:39vstinnersetmessages: + msg298033
2017-07-10 08:39:31vstinnersetmessages: + msg298032
2017-07-05 14:58:35serhiy.storchakasetmessages: + msg297763
2017-07-05 14:28:57vstinnersetmessages: + msg297762
2017-07-05 14:09:03belopolskysetmessages: + msg297760
2017-07-05 13:47:35vstinnersetmessages: + msg297753
2017-07-05 13:45:43vstinnersetmessages: + msg297752
2017-07-05 13:44:55vstinnersetmessages: + msg297743
2017-07-05 13:13:43serhiy.storchakasetmessages: + msg297736
2017-07-05 12:44:03vstinnersetpull_requests: + pull_request2658
2017-07-05 12:42:23vstinnersetmessages: + msg297734
2017-07-03 20:07:41vstinnersetmessages: + msg297612
2017-07-03 19:55:33musically_utsetpull_requests: + pull_request2621
2017-07-03 19:37:29musically_utsetpull_requests: + pull_request2620
2017-07-03 11:37:07serhiy.storchakasetmessages: + msg297573
2017-07-03 11:32:40musically_utsetmessages: + msg297571
2017-07-03 11:12:27vstinnersetmessages: + msg297562
2017-07-03 10:59:38musically_utsetmessages: + msg297559
2017-07-03 10:50:28vstinnersetmessages: + msg297557
2017-07-03 10:42:18vstinnersetmessages: + msg297555
2017-07-03 10:41:39vstinnersetnosy: + vstinner
messages: + msg297554
2017-07-03 09:01:38serhiy.storchakasetmessages: + msg297548
2017-07-03 08:33:40musically_utsetmessages: + msg297546
2017-07-03 04:09:14serhiy.storchakasetmessages: + msg297530
2017-07-02 19:37:07musically_utsetmessages: + msg297525
2017-07-02 19:36:14serhiy.storchakasetmessages: + msg297524
2017-07-02 18:07:16serhiy.storchakasetmessages: + msg297523
2017-07-02 18:02:36serhiy.storchakasetmessages: + msg297522
2017-07-02 17:55:20serhiy.storchakasetpull_requests: + pull_request2601
2017-07-02 17:14:56serhiy.storchakasetassignee: serhiy.storchaka

nosy: + belopolsky
2017-07-02 17:11:24musically_utsetmessages: + msg297520
2017-07-02 16:17:16serhiy.storchakasetmessages: + msg297519
2017-07-02 16:12:54ned.deilysetnosy: + ned.deily
messages: + msg297518
2017-07-02 13:52:47musically_utsetmessages: + msg297517
2017-07-02 12:47:03serhiy.storchakasetstage: needs patch -> backport needed
2017-07-02 12:46:06serhiy.storchakasetmessages: + msg297515
2017-07-01 23:44:36musically_utsetpull_requests: + pull_request2597
2017-07-01 04:45:07serhiy.storchakasettype: enhancement -> behavior
stage: needs patch
messages: + msg297470
versions: - Python 3.3, Python 3.4
2017-06-30 23:47:48musically_utcreate