classification
Title: test_datetime leaks memory on Windows
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: izbyshev, vstinner
Priority: normal Keywords: patch

Created on 2018-11-27 00:20 by vstinner, last changed 2018-11-28 11:47 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
datetimetester.py vstinner, 2018-11-27 00:20
datetimetester.py vstinner, 2018-11-27 00:24
Pull Requests
URL Status Linked Edit
PR 10759 merged vstinner, 2018-11-28 02:22
PR 10761 merged vstinner, 2018-11-28 10:34
Messages (9)
msg330482 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-27 00:20
test_datetime currently leaks memory blocks:

test_datetime leaked [3, 4, 3] memory blocks, sum=10

Attached patch should be added to Lib/test/ to only run a minimum set of tests which reproduce the bug.
msg330483 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-27 00:24
C:\vstinner\python\master>python -m test -R 3:20 -u all test_datetime
Running Debug|x64 interpreter...
Run tests sequentially
0:00:00 [1/1] test_datetime
beginning 23 repetitions
12345678901234567890123
.......................
test_datetime leaked [1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] memory blocks, sum=21
test_datetime failed

== Tests result: FAILURE ==

1 test failed:
    test_datetime

Total duration: 562 ms
Tests result: FAILURE
msg330534 - (view) Author: Alexey Izbyshev (izbyshev) * (Python triager) Date: 2018-11-27 16:15
This is because of a leak of 'wstr' at https://github.com/python/cpython/blob/1005c84535191a72ebb7587d8c5636a065b7ed79/Objects/unicodeobject.c#L3476 .

There is another leak and usage of uninitialized 'str' because the following "else if" clause doesn't have a 'return' (probably due to copy-paste from unicode_decode_locale() in 3d4226a832cabc630402589cc671cc4035d504e5).
msg330536 - (view) Author: Alexey Izbyshev (izbyshev) * (Python triager) Date: 2018-11-27 16:25
Correction: the fall-through in "else if (res == -3)" clause doesn't cause a memory leak, but still results in usage of uninitialized 'str'.
msg330571 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-28 02:28
Oh, I see. unicode_encode_locale() has multiple bugs in the code handling errors :-(

I don't think that my commit 3d4226a832cabc630402589cc671cc4035d504e5 introduced the memory leak triggered by test_datetime, it seems older. Python 3.7 also has the bug in unicode_encode_locale(). The memory leak has been introduced by commit 7ed7aead9503102d2ed316175f198104e0cd674c, also written by me :-)
msg330587 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-28 09:26
New changeset bde9d6bbb46ca59bcee5d5060adaa33c3ffee3a6 by Victor Stinner in branch 'master':
bpo-34523, bpo-35322: Fix unicode_encode_locale() (GH-10759)
https://github.com/python/cpython/commit/bde9d6bbb46ca59bcee5d5060adaa33c3ffee3a6
msg330589 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-28 10:36
bpo-34482 added the test in test_datetime which triggered the bug.
msg330592 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-28 11:42
New changeset 85ab974f78c0ebcfa611639864640d0273eb5466 by Victor Stinner in branch '3.7':
bpo-34523, bpo-35322: Fix unicode_encode_locale() (GH-10759) (GH-10761)
https://github.com/python/cpython/commit/85ab974f78c0ebcfa611639864640d0273eb5466
msg330593 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-28 11:47
I tested manually that "python -m test test_datetime -R 3:3" pass on 3.7 and master branches.

Note: PyUnicode_DecodeLocale() is still untested, only _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx() are tested.
History
Date User Action Args
2018-11-28 11:47:37vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg330593

stage: patch review -> resolved
2018-11-28 11:42:43vstinnersetmessages: + msg330592
2018-11-28 10:36:24vstinnersetmessages: + msg330589
2018-11-28 10:34:32vstinnersetstage: patch review
pull_requests: + pull_request10010
2018-11-28 09:26:24vstinnersetmessages: + msg330587
2018-11-28 02:28:06vstinnersetstage: patch review -> (no value)
messages: + msg330571
versions: + Python 3.7
2018-11-28 02:22:53vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request10007
2018-11-27 16:25:44izbyshevsetmessages: + msg330536
2018-11-27 16:15:29izbyshevsetnosy: + izbyshev
messages: + msg330534
2018-11-27 09:44:18vstinnersettitle: test_datetime leaks memory -> test_datetime leaks memory on Windows
components: + Library (Lib)
versions: + Python 3.8
2018-11-27 00:24:31vstinnersetmessages: + msg330483
2018-11-27 00:24:10vstinnersetfiles: + datetimetester.py
2018-11-27 00:20:52vstinnercreate