Make support.threading_cleanup() stricter #75417

vstinner · 2017-08-18T17:12:58Z

BPO	31234
Nosy	@vstinner
PRs	bpo-31234: threading_cleanup() now warns immediately #3138 bpo-31234: fork_wait tests now join threads #3139 bpo-31234: test_threading: fix ref cycle #3150 [3.6] bpo-31234: test_threading: fix ref cycle (#3150) #3152 [3.6] bpo-31234: fork_wait tests now join threads (#3139) #3187 bpo-31234: test_httpservers joins the server thread #3188 bpo-31234: test_threaded_import: fix test_side_effect_import() #3189 bpo-31234: Enhance test_thread.test_forkinthread() #3516 [3.6] bpo-31234: Enhance test_thread.test_forkinthread() (GH-3516) #3519 [2.7] bpo-31234: fork_wait tests now join threads (GH-3139) #3535 [3.6] bpo-31234: test_httpservers joins the server thread (GH-3188) #3536 [3.6] bpo-31234: test_threaded_import: fix test_side_effect_import() (GH-3189) #3537 bpo-31234: Fix dangling thread in test_ftp/poplib #3540 bpo-31234: Fix dangling thread in test_ftplib #3544 bpo-31234, socket.create_connection(): Fix ref cycle #3546 [3.6] bpo-31234, socket.create_connection(): Fix ref cycle (GH-3546) #3552 [3.6] bpo-31234: Fix dangling thread in test_ftp/poplib #3554 bpo-31234: Try to fix lock_tests warning #3557 bpo-31234: Join threads in tests #3572 bpo-31234: Join threads in test_hashlib #3573 bpo-31234: Add test.support.wait_threads_exit() #3578 bpo-31234: Join threads in test_threading #3579 bpo-31234: Join threads in test_queue #3586 bpo-31234: Add support.join_thread() helper #3587 [3.6] bpo-31234: Join threads in tests #3589 bpo-31234: Join timers in test_threading #3598 bpo-31234: test_multiprocessing: wait 30 seconds #3599 [3.6] bpo-31234: Add test.support.wait_threads_exit() (GH-3578) #7315 [2.7] bpo-31234: Join threads explicitly in tests #7406

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-10-17.20:39:07.246>
created_at = <Date 2017-08-18.17:12:58.276>
labels = ['3.7', 'tests']
title = 'Make support.threading_cleanup() stricter'
updated_at = <Date 2018-06-04.21:53:54.792>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-06-04.21:53:54.792>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-10-17.20:39:07.246>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2017-08-18.17:12:58.276>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 31234
keywords = ['patch']
message_count = 51.0
messages = ['300519', '300520', '300531', '300553', '300558', '300559', '300560', '300561', '300631', '300636', '300638', '300706', '300708', '300709', '301118', '301132', '301982', '302005', '302008', '302048', '302053', '302055', '302056', '302058', '302076', '302078', '302098', '302112', '302115', '302122', '302128', '302129', '302132', '302150', '302151', '302152', '302163', '302177', '302178', '302206', '302207', '302210', '302214', '302245', '302247', '302248', '302255', '302305', '304530', '318417', '318712']
nosy_count = 1.0
nosy_names = ['vstinner']
pr_nums = ['3138', '3139', '3150', '3152', '3187', '3188', '3189', '3516', '3519', '3535', '3536', '3537', '3540', '3544', '3546', '3552', '3554', '3557', '3572', '3573', '3578', '3579', '3586', '3587', '3589', '3598', '3599', '7315', '7406']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue31234'
versions = ['Python 3.7']

vstinner · 2017-08-18T17:12:58Z

Currently, support.threading_cleanup() waits 1 second before emitting a warning about threads running in the background. I propose to emit immediately the warning to be able to catch bugs quicker, bugs like bpo-31233 or bpo-30830.

The problem is that currently, 12 test files still leak threads:
---
12 tests altered the execution environment:
test_asyncio test_concurrent_futures test_fork1 test_httpservers
test_logging test_pydoc test_ssl test_thread test_threaded_import
test_threading test_wait3 test_xmlrpc
---

Since threading_cleanup() warnings now mark tests as failed, we should first fix these tests before being able to make support.threading_cleanup() stricter.

vstinner · 2017-08-18T17:19:13Z

test_fork1, test_wait3: #3139

vstinner · 2017-08-18T21:12:31Z

New changeset c99d41f by Victor Stinner in branch 'master':
bpo-31234: fork_wait tests now join threads (bpo-3139)
c99d41f

vstinner · 2017-08-18T23:32:41Z

test_threading: #3150

vstinner · 2017-08-18T23:53:10Z

test_pydoc: see bpo-31238.

vstinner · 2017-08-18T23:54:21Z

test_logging: see bpo-30830.

vstinner · 2017-08-18T23:54:45Z

New changeset 3d284c0 by Victor Stinner in branch 'master':
bpo-31234: test_threading: fix ref cycle (bpo-3150)
3d284c0

vstinner · 2017-08-19T00:32:57Z

New changeset b50e768 by Victor Stinner in branch '3.6':
bpo-31234: test_threading: fix ref cycle (bpo-3150) (bpo-3152)
b50e768

vstinner · 2017-08-21T15:53:23Z

test_xmlrpc: bpo-31247, "test_xmlrpc leaks dangling threads".

vstinner · 2017-08-21T16:16:22Z

test_concurrent_futures: bpo-31249, test_concurrent_futures leaks dangling threads.

vstinner · 2017-08-21T16:34:16Z

test_asyncio: bpo-31250, "test_asyncio leaks dangling threads".

vstinner · 2017-08-22T16:03:08Z

New changeset 5f20539 by Victor Stinner in branch '3.6':
bpo-31234: fork_wait tests now join threads (bpo-3139) (bpo-3187)
5f20539

vstinner · 2017-08-22T16:05:10Z

New changeset 830d7d2 by Victor Stinner in branch 'master':
bpo-31234: test_httpservers joins the server thread (bpo-3188)
830d7d2

vstinner · 2017-08-22T16:05:35Z

New changeset 41bbd82 by Victor Stinner in branch 'master':
bpo-31234: test_threaded_import: fix test_side_effect_import() (bpo-3189)
41bbd82

vstinner · 2017-09-01T15:02:00Z

test_ssl still emits a warning: see bpo-31323.

vstinner · 2017-09-01T16:00:26Z

Remaining issues:

test_asyncio: bpo-31250, test_asyncio: fix EventLoopTestsMixin.tearDown() #3264 of bpo-31250
test_thread: test_forkinthread() spawns a thread using thread.start_new_thread() without waiting until the thread completes
test_concurrent_futures: warnings raised when running "./python -u -m test -j0 -rW -u all --fail-env-changed", but not when running only test_concurrent_futures. It looks like a race condition.

vstinner · 2017-09-12T17:49:26Z

New changeset a15d155 by Victor Stinner in branch 'master':
bpo-31234: Enhance test_thread.test_forkinthread() (bpo-3516)
a15d155

vstinner · 2017-09-12T23:14:12Z

New changeset bcf042f by Victor Stinner (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-31234: Enhance test_thread.test_forkinthread() (GH-3516) (bpo-3519)
bcf042f

vstinner · 2017-09-13T00:04:03Z

Remaining issue: socketserver. bpo-31233: socketserver.ThreadingMixIn leaks running threads after server_close()

See also bpo-31151: socketserver.ForkingMixIn.server_close() leaks zombie processes.

vstinner · 2017-09-13T10:04:08Z

I rebased PR 3138: 3 tests failed on Travis CI.

test_ssl:

Warning -- threading_cleanup() failed to cleanup 0 threads (count: 0, dangling: 3)
Dangling thread: <HTTPSServerThread(Thread-209, stopped daemon 47903503230720)>
Dangling thread: <ThreadedEchoServer(Thread-173, stopped daemon 47903503230720)>
Dangling thread: <_MainThread(MainThread, started 47903383548928)>

test_poplib:

Warning -- threading_cleanup() failed to cleanup 0 threads (count: 0, dangling: 3)
Dangling thread: <test.test_poplib.DummyPOP3Server localhost:0 at 0x2b42fe691a80>
Dangling thread: <_MainThread(MainThread, started 47566631539712)>
Dangling thread: <test.test_poplib.DummyPOP3Server localhost:0 at 0x2b42fe6addc0>
/home/travis/build/python/cpython/Lib/test/support/init.py:1511: ResourceWarning: unclosed <socket.socket [closed] fd=5, family=AddressFamily.AF_INET, type=2049, proto=6>
gc.collect()
/home/travis/build/python/cpython/Lib/test/support/init.py:1511: ResourceWarning: unclosed <socket.socket [closed] fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
gc.collect()
/home/travis/build/python/cpython/Lib/test/support/init.py:1511: ResourceWarning: unclosed <socket.socket [closed] fd=7, family=AddressFamily.AF_INET, type=2049, proto=6>
gc.collect()

test_ftplib:

Warning -- threading_cleanup() failed to cleanup 0 threads (count: 0, dangling: 4)
Dangling thread: <test.test_ftplib.DummyTLS_FTPServer localhost:0 at 0x2b72275b4128>
Dangling thread: <test.test_ftplib.DummyTLS_FTPServer localhost:0 at 0x2b72279cb538>
Dangling thread: <Thread(Thread-41, stopped 47769290839808)>
Dangling thread: <_MainThread(MainThread, started 47769182004224)>

vstinner · 2017-09-13T10:26:56Z

New changeset 42f7e0d by Victor Stinner in branch '2.7':
bpo-31234: fork_wait tests now join threads (bpo-3139) (bpo-3535)
42f7e0d

vstinner · 2017-09-13T10:27:47Z

New changeset 89bfc9b by Victor Stinner (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-31234: test_threaded_import: fix test_side_effect_import() (GH-3189) (bpo-3537)
89bfc9b

vstinner · 2017-09-13T10:27:56Z

New changeset 6c25b73 by Victor Stinner (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-31234: test_httpservers joins the server thread (GH-3188) (bpo-3536)
6c25b73

vstinner · 2017-09-13T10:58:28Z

New changeset d403a29 by Victor Stinner in branch 'master':
bpo-31234: Fix dangling thread in test_ftp/poplib (bpo-3540)
d403a29

vstinner · 2017-09-13T13:44:00Z

New changeset b157ce1 by Victor Stinner in branch 'master':
bpo-31234: Fix dangling thread in test_ftplib (bpo-3544)
b157ce1

vstinner · 2017-09-13T14:11:04Z

test_ssl:

test_recv_zero (test.test_ssl.ThreadedTests) ... server: new connection from ('127.0.0.1', 54502)
server: connection cipher is now ('ECDHE-RSA-AES256-SHA', 'TLSv1/SSLv3', 256)
server: selected protocol is now None
Warning -- threading_cleanup() failed to cleanup 0 threads (count: 0, dangling: 2)
Dangling thread: <_MainThread(MainThread, started 47577226880000)>
Dangling thread: <ThreadedEchoServer(Thread-173, stopped daemon 47577348138752)>
ok

test_socketserver (test.test_ssl.ThreadedTests)
(...)
stopping HTTPS server
joining HTTPS thread
Warning -- threading_cleanup() failed to cleanup 0 threads (count: 0, dangling: 2)
Dangling thread: <HTTPSServerThread(Thread-209, stopped daemon 47577348138752)>
Dangling thread: <_MainThread(MainThread, started 47577226880000)>
ok

vstinner · 2017-09-13T17:10:13Z

New changeset acb9fa7 by Victor Stinner in branch 'master':
bpo-31234, socket.create_connection(): Fix ref cycle (bpo-3546)
acb9fa7

vstinner · 2017-09-13T20:42:03Z

New changeset ace1ecc by Victor Stinner in branch 'master':
bpo-31234: threading_cleanup() now warns immediately (bpo-3138)
ace1ecc

vstinner · 2017-09-13T21:02:08Z

Hum, test_barrier of test_thread failed:

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%203.x/builds/880/steps/test/logs/stdio

0:02:09 load avg: 2.50 [106/405/1] test_thread failed (env changed) -- running: test_multiprocessing_spawn (63 sec), test_multiprocessing_forkserver (54 sec)

test_barrier (test.test_thread.BarrierTest) ...
(...)
task 8 leaving barrier
task 2 leaving barrier
task 9 leaving barrier
tasks done
Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 1)
Dangling thread: <_MainThread(MainThread, started 34393318400)>
ok

vstinner · 2017-09-13T21:41:03Z

Source_ModuleLockAsRLockTests.test_different_thread() of test.test_importlib.test_locks failed:

http://buildbot.python.org/all/builders/x86-64%20El%20Capitan%203.x/builds/758/steps/test/logs/stdio

test_different_thread (test.test_importlib.test_locks.Source_ModuleLockAsRLockTests) ... Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 1)
Dangling thread: <_MainThread(MainThread, started 140735235059712)>
ok

vstinner · 2017-09-13T22:54:39Z

New changeset d99e85b by Victor Stinner (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-31234, socket.create_connection(): Fix ref cycle (GH-3546) (bpo-3552)
d99e85b

vstinner · 2017-09-13T22:56:17Z

New changeset ef323e8 by Victor Stinner in branch '3.6':
[3.6] bpo-31234: Fix dangling thread in test_ftp/poplib (bpo-3554)
ef323e8

vstinner · 2017-09-13T23:41:13Z

New changeset 096ae33 by Victor Stinner in branch 'master':
bpo-31234: Try to fix lock_tests warning (bpo-3557)
096ae33

vstinner · 2017-09-14T07:25:08Z

test_threading.test_foreign_thread() failed:

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%20CURRENT%20Debug%203.x/builds/879/steps/test/logs/stdio

0:08:53 load avg: 1.84 [286/405/1] test_threading failed (env changed) -- running: test_multiprocessing_spawn (51 sec)
(...)
test_dummy_thread_after_fork (test.test_threading.ThreadTests) ... ok
test_enumerate_after_join (test.test_threading.ThreadTests) ... ok
test_finalize_runnning_thread (test.test_threading.ThreadTests) ... ok
test_finalize_with_trace (test.test_threading.ThreadTests) ... ok
test_foreign_thread (test.test_threading.ThreadTests) ... Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 1)
Dangling thread: <_MainThread(MainThread, started 34369060864)>
ok
test_frame_tstate_tracing (test.test_threading.ThreadTests) ... ok

vstinner · 2017-09-14T07:27:47Z

3 more warnings in test_threading on AMD64 FreeBSD 10.x Shared 3.x:

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%203.x/builds/886/steps/test/logs/stdio

0:03:45 load avg: 4.05 [246/405/1] test_thread failed (env changed)
(...)
test_nt_and_posix_stack_size (test.test_thread.ThreadRunningTests) ... (...)
task 4 done
task 10 done
all tasks done
Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 1)
Dangling thread: <_MainThread(MainThread, started 34393318400)>
ok
(...)
test_starting_threads (test.test_thread.ThreadRunningTests) ...
(...)
task 9 will run for 55us
task 9 done
all tasks done
Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 1)
Dangling thread: <_MainThread(MainThread, started 34393318400)>
ok

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%203.x/builds/884/steps/test/logs/stdio

0:05:44 load avg: 3.00 [249/405/1] test_thread failed (env changed) -- running: test_concurrent_futures (31 sec), test_multiprocessing_fork (60 sec)
test_barrier (test.test_thread.BarrierTest) ...
(...)
task 5 leaving barrier
task 3 leaving barrier
tasks done
Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 1)
Dangling thread: <_MainThread(MainThread, started 34393318400)>
ok

vstinner · 2017-09-14T07:29:02Z

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%20CURRENT%20Non-Debug%203.x/builds/881/steps/test/logs/stdio

0:08:05 load avg: 1.58 [288/405/1] test_threading failed (env changed) -- running: test_multiprocessing_spawn (83 sec)
(...)
test_dummy_thread_after_fork (test.test_threading.ThreadTests) ... ok
test_enumerate_after_join (test.test_threading.ThreadTests) ... ok
test_finalize_runnning_thread (test.test_threading.ThreadTests) ... ok
test_finalize_with_trace (test.test_threading.ThreadTests) ... ok
test_foreign_thread (test.test_threading.ThreadTests) ...

Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 1)
Dangling thread: <_MainThread(MainThread, started 34368585728)>

ok
test_frame_tstate_tracing (test.test_threading.ThreadTests) ... ok
test_ident_of_no_threading_threads (test.test_threading.ThreadTests) ... ok

vstinner · 2017-09-14T09:59:32Z

Oh, test_concurrent_futures.test_exception_with_success() failed on Travis CI :-(

https://ci.appveyor.com/project/python/cpython/build/3.7.0a0.6457

test_exception_with_success (test.test_concurrent_futures.FutureTests) ... Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 2)
Dangling thread: <Thread(Thread-1, started 4012)>
Dangling thread: <_MainThread(MainThread, started 3656)>
ok

vstinner · 2017-09-14T15:43:09Z

New changeset 18e95b4 by Victor Stinner in branch 'master':
bpo-31234: Join threads in tests (bpo-3572)
18e95b4

vstinner · 2017-09-14T15:43:25Z

New changeset 8dcf22f by Victor Stinner in branch 'master':
bpo-31234: Join threads in test_hashlib (bpo-3573)
8dcf22f

vstinner · 2017-09-14T20:05:24Z

New changeset b8c7be2 by Victor Stinner in branch 'master':
bpo-31234: Join threads in test_threading (bpo-3579)
b8c7be2

vstinner · 2017-09-14T20:07:26Z

New changeset ff40ecd by Victor Stinner in branch 'master':
bpo-31234: Add test.support.wait_threads_exit() (bpo-3578)
ff40ecd

vstinner · 2017-09-14T21:04:58Z

New changeset 167cbde by Victor Stinner in branch 'master':
bpo-31234: Join threads in test_queue (bpo-3586)
167cbde

vstinner · 2017-09-14T21:40:58Z

New changeset b9b6900 by Victor Stinner in branch 'master':
bpo-31234: Add support.join_thread() helper (bpo-3587)
b9b6900

vstinner · 2017-09-15T11:46:28Z

Buildbots want my death :-( They never stop failing. Yet another random warning considered as an error:

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%20CURRENT%20Non-Debug%203.x/builds/896/steps/test/logs/stdio

Run tests in parallel using 2 child processes
(...)
0:07:33 load avg: 1.17 [285/405/1] test_threading failed (env changed)
(...)
test_init_immutable_default_args (test.test_threading.TimerTests) ... Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 2)
Dangling thread: <Timer(Thread-261, started 34397998592)>
Dangling thread: <_MainThread(MainThread, started 34368581632)>
ok

vstinner · 2017-09-15T12:37:44Z

New changeset da3e5cf by Victor Stinner in branch 'master':
bpo-31234: Join timers in test_threading (bpo-3598)
da3e5cf

vstinner · 2017-09-15T12:39:30Z

Oh, one change of the commit b9b6900 introduced a regression x86 Gentoo Refleaks 3.x. It seems like 5 seconds is not enough on this slow buildbot.

http://buildbot.python.org/all/builders/x86%20Gentoo%20Refleaks%203.x/builds/93/steps/test/logs/stdio

2:28:30 load avg: 6.98 [263/405/1] test_multiprocessing_spawn failed -- running: test_zipfile (797 sec)
beginning 6 repetitions
123456
.Process Process-555:1:1:
Traceback (most recent call last):
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.refleak/build/Lib/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.refleak/build/Lib/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.refleak/build/Lib/test/_test_multiprocessing.py", line 4050, in child
    join_process(p, timeout=5)
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.refleak/build/Lib/test/_test_multiprocessing.py", line 79, in join_process
    support.join_thread(process, timeout)
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.refleak/build/Lib/test/support/__init__.py", line 2117, in join_thread
    raise AssertionError(msg)
AssertionError: failed to join the thread in 5.0 seconds

vstinner · 2017-09-15T13:55:33Z

New changeset 11f0807 by Victor Stinner in branch 'master':
bpo-31234: test_multiprocessing: wait 30 seconds (bpo-3599)
11f0807

vstinner · 2017-09-15T23:55:54Z

New changeset 2c1c2ca by Victor Stinner in branch '3.6':
[3.6] bpo-31234: Join threads in tests (bpo-3589)
2c1c2ca

vstinner · 2017-10-17T20:39:07Z

It seems like all attached PR are now merged. I didn't see any random "dangling thread" warning recently in the master branch, so I close this issue.

vstinner · 2018-06-01T13:51:05Z

New changeset 5dbb48a by Victor Stinner in branch '3.6':
[3.6] bpo-31234: Add test.support.wait_threads_exit() (GH-3578) (GH-7315)
5dbb48a

vstinner · 2018-06-04T21:53:55Z

New changeset 1463518 by Victor Stinner in branch '2.7':
[2.7] bpo-31234: Join threads explicitly in tests (bpo-7406)
1463518

vstinner added 3.7 (EOL) end of life tests Tests in the Lib/test dir labels Aug 18, 2017

vstinner closed this as completed Oct 17, 2017

ezio-melotti transferred this issue from another repository Apr 10, 2022

Make support.threading_cleanup() stricter #75417

Make support.threading_cleanup() stricter #75417

Comments

vstinner commented Aug 18, 2017

vstinner commented Aug 18, 2017

vstinner commented Aug 18, 2017

vstinner commented Aug 18, 2017

vstinner commented Aug 18, 2017

vstinner commented Aug 18, 2017

vstinner commented Aug 18, 2017

vstinner commented Aug 18, 2017

vstinner commented Aug 19, 2017

vstinner commented Aug 21, 2017

vstinner commented Aug 21, 2017

vstinner commented Aug 21, 2017

vstinner commented Aug 22, 2017

vstinner commented Aug 22, 2017

vstinner commented Aug 22, 2017

vstinner commented Sep 1, 2017

vstinner commented Sep 1, 2017

vstinner commented Sep 12, 2017

vstinner commented Sep 12, 2017

vstinner commented Sep 13, 2017

vstinner commented Sep 13, 2017

vstinner commented Sep 13, 2017

vstinner commented Sep 13, 2017

vstinner commented Sep 13, 2017

vstinner commented Sep 13, 2017

vstinner commented Sep 13, 2017

vstinner commented Sep 13, 2017

vstinner commented Sep 13, 2017

vstinner commented Sep 13, 2017

vstinner commented Sep 13, 2017

vstinner commented Sep 13, 2017

vstinner commented Sep 13, 2017

vstinner commented Sep 13, 2017

vstinner commented Sep 13, 2017

vstinner commented Sep 14, 2017

vstinner commented Sep 14, 2017

vstinner commented Sep 14, 2017

vstinner commented Sep 14, 2017

vstinner commented Sep 14, 2017

vstinner commented Sep 14, 2017

vstinner commented Sep 14, 2017

vstinner commented Sep 14, 2017

vstinner commented Sep 14, 2017

vstinner commented Sep 14, 2017

vstinner commented Sep 15, 2017

vstinner commented Sep 15, 2017

vstinner commented Sep 15, 2017

vstinner commented Sep 15, 2017

vstinner commented Sep 15, 2017

vstinner commented Oct 17, 2017

vstinner commented Jun 1, 2018

vstinner commented Jun 4, 2018