msg202414 - (view) |
Author: Bohuslav "Slavek" Kabrda (bkabrda) * |
Date: 2013-11-08 12:30 |
When Python is compiled with COUNT_ALLOCS, some tests in test_gc and test_module fail. I'm attaching the patch that skips 3 of them and modifies assertions in one of them, so that the tests pass.
I'm however still unsure about one of the skipped tests, since I'm unsure whether I totally understand what's wrong there - test_gc_ordinary_module_at_shutdown.
My guess is that due to COUNT_ALLOCS causing immortal types, the "final_a" and "final_b" types don't get destroyed on line [1] as they do in builds without COUNT_ALLOCS. AFAICS they are only "un-immortalized" on this line and destroyed during the following loop [2]. The problem here is that the order of destroyed modules is not deterministic, so sometimes the builtins module gets destroyed before the "final_X" and there is no "print" function, which makes the __del__ functions from "final_X" fail. IMO the best thing to do is to just skip this test with COUNT_ALLOCS. But I may be wrong, I don't have a great insight into Python's GC and module unloading.
[1] http://hg.python.org/cpython/annotate/0f48843652b1/Python/import.c#l383
[2] http://hg.python.org/cpython/annotate/0f48843652b1/Python/import.c#l394
|
msg202415 - (view) |
Author: Bohuslav "Slavek" Kabrda (bkabrda) * |
Date: 2013-11-08 12:30 |
And the patch...
|
msg207844 - (view) |
Author: Bohuslav "Slavek" Kabrda (bkabrda) * |
Date: 2014-01-10 10:25 |
Since 3.4.0.b2, this also causes failures in another tests: test_io, test_logging, test_threading, test_warnings. There are various cases testing that some types get collected when the interpreter shuts down.
I'm attaching a new patch that covers all of these.
|
msg207845 - (view) |
Author: STINNER Victor (vstinner) *  |
Date: 2014-01-10 10:28 |
What is COUNT_ALLOCS?
Python 3.4 now has sys.getallocatedblocks() and a new tracemalloc module which are compiled by default.
|
msg207848 - (view) |
Author: Bohuslav "Slavek" Kabrda (bkabrda) * |
Date: 2014-01-10 10:33 |
As noted in Misc/SpecialBuilds:
COUNT_ALLOCS
------------
Each type object grows three new members:
/* Number of times an object of this type was allocated. */
int tp_allocs;
/* Number of times an object of this type was deallocated. */
int tp_frees;
/* Highwater mark: the maximum value of tp_allocs - tp_frees so
* far; or, IOW, the largest number of objects of this type alive at
* the same time.
*/
int tp_maxalloc;
...
We use this for Fedora's python debug build to get some interesting debugging info. (If you try to "grep -r" through Python 3.4 source code, you'll find quite a few ifdefs with COUNT_ALLOCS.)
|
msg232450 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) *  |
Date: 2014-12-10 22:40 |
Why do you check hasattr(sys, 'getrefcount') in test_io.py, but hasattr(sys, 'getcounts') in all other tests?
|
msg232469 - (view) |
Author: Bohuslav "Slavek" Kabrda (bkabrda) * |
Date: 2014-12-11 11:22 |
Good catch, using getrefcount was a mistake. I'm attaching a new version which always checks for getcounts (and also applies on 3.4.2).
|
msg232492 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) *  |
Date: 2014-12-11 20:55 |
LGTM (only one nitpick -- there are trailing spaces in test_gc).
But there are other tests which are failed with COUNT_ALLOCS. Here is a patch with additional fixes for these tests in test_gc and test_warnings.
|
msg232615 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) *  |
Date: 2014-12-13 17:47 |
Thenks Antoine for great idea proposed in comments on Rietveld. Following patch introduces strip_python_stdout() which strips COUNT_ALLOCS debug output from stdout (unfortunately this operation is not always unambiguous) and call it in assert_python_ok() and assert_python_failure(). This automatically fixes a large number of tests. Also fixed a number of other tests failing with COUNT_ALLOCS. Virtually all tests are now fixed except test_doctest and test_distutils.
A large part of the patch will be applied only to 3.4 and 2.7. With resolved issue23034 the patch for 3.5 will be much simpler, strip_python_stdout() and @requires_clean_stdout will gone away.
|
msg255697 - (view) |
Author: Robert Kuska (rkuska) * |
Date: 2015-12-02 10:03 |
With Python-3.5 and COUNT_ALLOCS enabled following new tests fail also:
FAIL: test_is_finalizing (test.test_sys.SysModuleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/builddir/build/BUILD/Python-3.5.0/Lib/test/test_sys.py", line 767, in test_is_finalizing
self.assertEqual(stdout.rstrip(), b'True')
AssertionError: b'' != b'True'
---------------------------------
======================================================================
FAIL: test_print_traceback_at_exit (test.test_traceback.SyntaxTracebackCases)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/builddir/build/BUILD/Python-3.5.0/Lib/test/test_traceback.py", line 210, in test_print_traceback_at_exit
self.assertEqual(stderr.splitlines(), expected)
AssertionError: Lists differ: [] != [b'Traceback (most recent call last):', b'[75 chars]ero']
Second list contains 3 additional elements.
First extra element 0:
b'Traceback (most recent call last):'
- []
+ [b'Traceback (most recent call last):',
+ b' File "<string>", line 8, in __init__',
+ b'ZeroDivisionError: division by zero']
|
msg255698 - (view) |
Author: STINNER Victor (vstinner) *  |
Date: 2015-12-02 10:28 |
COUNT_ALLOCS was added 22 years ago. I guess that the usecase is to track memory leaks, right?
branch: legacy-trunk
user: Sjoerd Mullender <sjoerd@acm.org>
date: Mon Oct 11 12:54:31 1993 +0000
files: Include/object.h Modules/arraymodule.c Modules/config.c.in Modules/imageop.c Modules/imgfile.c Objects/floatobject.c Objects/intobject.c Objects/listobj
description:
* Extended X interface: pixmap objects, colormap objects visual objects,
image objects, and lots of new methods.
* Added counting of allocations and deallocations of builtin types if
COUNT_ALLOCS is defined. Had to move calls to NEWREF down in some
files.
* Bug fix in sorting lists.
|
msg255700 - (view) |
Author: STINNER Victor (vstinner) *  |
Date: 2015-12-02 10:33 |
00141-fix-tests_with_COUNT_ALLOCS-v5.patch: please don't do that! It makes tests much more verbose for a compilation option which is hidden and probably almost never used in the wild. The option has no configuration option for example.
*If* you really want to keep the feature, I would prefer to make it more visible (add a configuration option) and disable the output at exit by default. It's better to add a new "-X showcountallocs" as it was done with "-X showrefcount". Before, we had to fix a lot of unit tests (as 00141-fix-tests_with_COUNT_ALLOCS-v5.patch) to strip the "[xxx refs]" from stderr, it was very annoying.
"Python 3.4 now has sys.getallocatedblocks() and a new tracemalloc module which are compiled by default."
IMHO these two debug features superseded COUNT_ALLOCS. Please try to convince me of the use case of this very old debug feature.
|
msg255701 - (view) |
Author: Robert Kuska (rkuska) * |
Date: 2015-12-02 10:39 |
FYI There is also issue23034 where is proposed "-X showalloccount" option to suppress the output, we use (custom patch) environment variable to filter out the verbose output in Fedora.
|
msg255702 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) *  |
Date: 2015-12-02 10:46 |
Yes, I also don't want to use 00141-fix-tests_with_COUNT_ALLOCS-v5.patch if there is better alternative. See issue23034 (I'm uncertain only in option name).
|
msg255703 - (view) |
Author: Bohuslav "Slavek" Kabrda (bkabrda) * |
Date: 2015-12-02 10:47 |
> IMHO these two debug features superseded COUNT_ALLOCS. Please try to convince me of the use case of this very old debug feature.
I no longer use this feature and I think that noone does. As you said, it seems to be obsoleted by other new features, so my vote would be to remove COUNT_ALLOCS altogether.
|
msg255711 - (view) |
Author: STINNER Victor (vstinner) *  |
Date: 2015-12-02 12:00 |
I propose to emit a compiler warning (or even an error?) in 3.5.x and drop
the code in 3.6. I don't think that a long deprecation period is requied.
The feature is not widely used.
|
msg269772 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2016-07-03 19:31 |
New changeset 5abf6cdcac4d by Serhiy Storchaka in branch '3.5':
Issue #19527: Fixed tests with defined COUNT_ALLOCS.
https://hg.python.org/cpython/rev/5abf6cdcac4d
New changeset e7d84ecdd37d by Serhiy Storchaka in branch 'default':
Issue #19527: Fixed tests with defined COUNT_ALLOCS.
https://hg.python.org/cpython/rev/e7d84ecdd37d
|
msg269791 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) *  |
Date: 2016-07-04 16:33 |
All tests now are passed in 3.6 on Linux. Making them passing in 3.5 requires too much changes that are not needed in 3.6. I don't think we need to pollute tests with these temporary workarounds.
|
msg304492 - (view) |
Author: STINNER Victor (vstinner) *  |
Date: 2017-10-17 09:25 |
New changeset 7b4ba62e388474e811268322b47f80d464933541 by Victor Stinner in branch '2.7':
[2.7] bpo-31692: Add PYTHONSHOWALLOCCOUNT env var (GH-3927)
https://github.com/python/cpython/commit/7b4ba62e388474e811268322b47f80d464933541
|
|
Date |
User |
Action |
Args |
2022-04-11 14:57:53 | admin | set | github: 63726 |
2017-10-17 09:25:27 | vstinner | set | messages:
+ msg304492 |
2017-10-09 09:21:52 | vstinner | set | pull_requests:
+ pull_request3902 |
2017-10-06 19:58:09 | vstinner | set | pull_requests:
+ pull_request3883 |
2016-07-04 16:33:17 | serhiy.storchaka | set | status: open -> closed versions:
+ Python 3.6, - Python 3.4 messages:
+ msg269791
resolution: fixed stage: patch review -> resolved |
2016-07-03 19:31:16 | python-dev | set | nosy:
+ python-dev messages:
+ msg269772
|
2015-12-02 12:00:03 | vstinner | set | messages:
+ msg255711 |
2015-12-02 10:47:19 | bkabrda | set | messages:
+ msg255703 |
2015-12-02 10:46:25 | serhiy.storchaka | set | messages:
+ msg255702 |
2015-12-02 10:39:35 | rkuska | set | messages:
+ msg255701 |
2015-12-02 10:33:16 | vstinner | set | messages:
+ msg255700 |
2015-12-02 10:28:45 | vstinner | set | messages:
+ msg255698 |
2015-12-02 10:03:31 | rkuska | set | nosy:
+ rkuska messages:
+ msg255697
|
2014-12-13 17:47:36 | serhiy.storchaka | set | files:
+ 00141-fix-tests_with_COUNT_ALLOCS-v5.patch
dependencies:
+ Dynamically control debugging output messages:
+ msg232615 |
2014-12-11 20:55:01 | serhiy.storchaka | set | files:
+ 00141-fix-tests_with_COUNT_ALLOCS-v4.patch
messages:
+ msg232492 |
2014-12-11 12:07:34 | serhiy.storchaka | set | assignee: serhiy.storchaka |
2014-12-11 11:22:03 | bkabrda | set | files:
+ 00141-fix-tests_with_COUNT_ALLOCS-v3.patch
messages:
+ msg232469 |
2014-12-10 22:40:04 | serhiy.storchaka | set | messages:
+ msg232450 |
2014-12-10 22:35:46 | serhiy.storchaka | set | versions:
+ Python 3.5 nosy:
+ serhiy.storchaka
components:
+ Tests type: behavior stage: patch review |
2014-01-10 10:33:30 | bkabrda | set | messages:
+ msg207848 |
2014-01-10 10:28:02 | vstinner | set | messages:
+ msg207845 |
2014-01-10 10:25:22 | bkabrda | set | files:
+ 00141-fix-tests_with_COUNT_ALLOCS-v2.patch
messages:
+ msg207844 |
2013-11-08 12:37:47 | vstinner | set | nosy:
+ vstinner
|
2013-11-08 12:30:50 | bkabrda | set | files:
+ 00141-fix-tests_with_COUNT_ALLOCS.patch keywords:
+ patch messages:
+ msg202415
|
2013-11-08 12:30:13 | bkabrda | create | |