I would like to report memory leaks in Python 3.6.3 when embedded in a C++ Windows app. I have download the Python 3.6.3 code and compiled a debug version of Python with my copy of Visual Studio 2017 Professional (have also confirmed the problem in Visual Studio 2015 Professional) and built the debug version of the Python DLL. With the following sample app I get memory leaks when compiled in debug mode:


//Pull in the Windows header and the MSVC Debug C Runtime
#include <Windows.h>
#define _CRTDBG_MAP_ALLOC
#include <stdlib.h>
#include <crtdbg.h>

//Pull in Python
#include <Python.h> 

int __stdcall WinMain(HINSTANCE /*hInstance*/, HINSTANCE /*hPrevInstance*/, _In_ LPTSTR /*lpCmdLine*/, int /*nCmdShow*/)
{
  //Track memory leaks using the MSVC Debug CRT
  _CrtSetDbgFlag(_CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF);
  //_CrtSetBreakAlloc(81);

  //Initialize Python
  Py_Initialize();
  PyEval_InitThreads();

  //Pretend this app does something!
  Sleep(2000);

  //Shutdown Python
  Py_FinalizeEx();
}

Taking out the calls to Py_Initialize, PyEval_InitThreads & Py_FinalizeEx results in no memory leaks being reported. The actual reported memory leaks are reported to the output window in Visual Studio as follows:

Detected memory leaks!
Dumping objects ->
{700} normal block at 0x014E0078, 2604 bytes long.
Data: <    r       o   > 00 00 0A 1C 72 FB FB FB 00 00 0A 0C 6F FB FB FB
{ 698 } normal block at 0x01502148, 2604 bytes long.
Data : <    r       o   > 00 00 0A 1C 72 FB FB FB 00 00 0A 0C 6F FB FB FB
{ 697 } normal block at 0x014DA350, 52 bytes long.
Data : <   $r           > 00 00 00 24 72 FB FB FB FF FF FF FF FF FF FF FF
{ 458 } normal block at 0x01511A88, 81968 bytes long.
Data : <  @ r     @ o   > 00 01 40 20 72 FB FB FB 00 01 40 10 6F FB FB FB
{ 389 } normal block at 0x014F1270, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 359 } normal block at 0x014FA730, 5168 bytes long.
Data : <    r       o   > 00 00 14 20 72 FB FB FB 00 00 14 10 6F FB FB FB
{ 319 } normal block at 0x014EE120, 2604 bytes long.
Data : <    r       o   > 00 00 0A 1C 72 FB FB FB 00 00 0A 0C 6F FB FB FB
{ 315 } normal block at 0x014EC478, 1200 bytes long.
Data : <    r       o   > 00 00 04 A0 72 FB FB FB 00 00 04 90 6F FB FB FB
{ 307 } normal block at 0x01500B98, 529 bytes long.
Data : <    r       o   > 00 00 02 01 72 FB FB FB 00 00 01 F1 6F FB FB FB
{ 281 } normal block at 0x01500350, 709 bytes long.
Data : <    r       o   > 00 00 02 B5 72 FB FB FB 00 00 02 A5 6F FB FB FB
{ 264 } normal block at 0x014FDAF8, 10284 bytes long.
Data : <  (r(o   > 00 00 28 1C 72 FB FB FB 00 00 28 0C 6F FB FB FB
{ 253 } normal block at 0x014F89F0, 539 bytes long.
Data : <    r       o   > 00 00 02 0B 72 FB FB FB 00 00 01 FB 6F FB FB FB
{ 248 } normal block at 0x014F5448, 847 bytes long.
Data: < ? r / o   > 00 00 03 3F 72 FB FB FB 00 00 03 2F 6F FB FB FB
{ 247 } normal block at 0x014F0A78, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 235 } normal block at 0x014F2A58, 1409 bytes long.
Data : <   qr      ao   > 00 00 05 71 72 FB FB FB 00 00 05 61 6F FB FB FB
{ 234 } normal block at 0x014F1FB8, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 233 } normal block at 0x014F1A68, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 232 } normal block at 0x014F1518, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 231 } normal block at 0x014E2BE0, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 230 } normal block at 0x014EFFF8, 2604 bytes long.
Data : <    r       o   > 00 00 0A 1C 72 FB FB FB 00 00 0A 0C 6F FB FB FB
{ 229 } normal block at 0x014EFD60, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 228 } normal block at 0x014EF9E8, 843 bytes long.
Data : <; r + o   > 00 00 03 3B 72 FB FB FB 00 00 03 2B 6F FB FB FB
{ 227 } normal block at 0x014EF750, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 226 } normal block at 0x014E4410, 561 bytes long.
Data : <   !r       o   > 00 00 02 21 72 FB FB FB 00 00 02 11 6F FB FB FB
{ 225 } normal block at 0x014EF1D0, 1360 bytes long.
Data : <   @r      0o   > 00 00 05 40 72 FB FB FB 00 00 05 30 6F FB FB FB
{ 224 } normal block at 0x014E49E0, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 206 } normal block at 0x014E3C88, 1200 bytes long.
Data : <    r       o   > 00 00 04 A0 72 FB FB FB 00 00 04 90 6F FB FB FB
{ 205 } normal block at 0x014E3978, 734 bytes long.
Data : <    r       o   > 00 00 02 CE 72 FB FB FB 00 00 02 BE 6F FB FB FB
{ 204 } normal block at 0x014E3358, 1521 bytes long.
Data : <    r       o   > 00 00 05 E1 72 FB FB FB 00 00 05 D1 6F FB FB FB
{ 195 } normal block at 0x014E9890, 1553 bytes long.
Data : <    r       o   > 00 00 06 01 72 FB FB FB 00 00 05 F1 6F FB FB FB
{ 192 } normal block at 0x014E8648, 734 bytes long.
Data : <    r       o   > 00 00 02 CE 72 FB FB FB 00 00 02 BE 6F FB FB FB
{ 191 } normal block at 0x014E8508, 276 bytes long.
Data : <    r   C : \ p > 00 00 01 04 72 FB FB FB 43 00 3A 00 5C 00 70 00
{118} normal block at 0x014E2948, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 117 } normal block at 0x014E26B0, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 116 } normal block at 0x014E1F38, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 115 } normal block at 0x014E21D0, 1200 bytes long.
Data : <    r       o   > 00 00 04 A0 72 FB FB FB 00 00 04 90 6F FB FB FB
{ 113 } normal block at 0x014E1CA0, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 111 } normal block at 0x014E0DB0, 593 bytes long.
Data : <   Ar      1o   > 00 00 02 41 72 FB FB FB 00 00 02 31 6F FB FB FB
{ 110 } normal block at 0x014E1528, 1200 bytes long.
Data : <    r       o   > 00 00 04 A0 72 FB FB FB 00 00 04 90 6F FB FB FB
{ 108 } normal block at 0x014E1048, 1200 bytes long.
Data : <    r       o   > 00 00 04 A0 72 FB FB FB 00 00 04 90 6F FB FB FB
{ 106 } normal block at 0x014E0B18, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 105 } normal block at 0x014DD3C0, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 104 } normal block at 0x014DD128, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 103 } normal block at 0x014DCE90, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 102 } normal block at 0x014DCBF8, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 101 } normal block at 0x014DC9B8, 532 bytes long.
Data : <    r       o   > 00 00 02 04 72 FB FB FB 00 00 01 F4 6F FB FB FB
{ 99 } normal block at 0x014DEF40, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 98 } normal block at 0x014DF1D8, 1200 bytes long.
Data : <    r       o   > 00 00 04 A0 72 FB FB FB 00 00 04 90 6F FB FB FB
{ 96 } normal block at 0x014DD658, 577 bytes long.
Data : <   1r      !o   > 00 00 02 31 72 FB FB FB 00 00 02 21 6F FB FB FB
{ 95 } normal block at 0x014DEA60, 1200 bytes long.
Data : <    r       o   > 00 00 04 A0 72 FB FB FB 00 00 04 90 6F FB FB FB
{ 93 } normal block at 0x014DE790, 670 bytes long.
Data : <    r      ~o   > 00 00 02 8E 72 FB FB FB 00 00 02 7E 6F FB FB FB
{ 92 } normal block at 0x014DE2B0, 1200 bytes long.
Data : <    r       o   > 00 00 04 A0 72 FB FB FB 00 00 04 90 6F FB FB FB
{ 90 } normal block at 0x014DDDD0, 1200 bytes long.
Data : <    r       o   > 00 00 04 A0 72 FB FB FB 00 00 04 90 6F FB FB FB
{ 88 } normal block at 0x014DD8F0, 1200 bytes long.
Data : <    r       o   > 00 00 04 A0 72 FB FB FB 00 00 04 90 6F FB FB FB
{ 86 } normal block at 0x014DC720, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 85 } normal block at 0x014DC488, 620 bytes long.
Data : <   \r      Lo   > 00 00 02 5C 72 FB FB FB 00 00 02 4C 6F FB FB FB
{ 81 } normal block at 0x014DAC08, 464 bytes long.
Data : <    r : > > 00 00 01 C0 72 FB FB FB 00 00 3A 03 00 00 3E 03
Object dump complete.

If you uncomment the call to _CrtSetBreakAlloc and set the parameter to this API to for example 81, the Visual Studio debugger will break where the memory allocation occurs. I have taken all the numbers and confirmed these are memory allocations which occur in the Python functions I am using. Most of these are due to memory allocations of static C variables in the Python source code.

I would be most grateful if these issues can be fixed up in the Python codebase. I am available to answer any questions about the issue at any time if necessary.

Regards,
PJ Naughter

Not sure how much we can do about this, but you may want to try with Python 3.7. Eric and Nick have been working on streamlining startup, which includes a lot of work (thanks Eric!) to remove static variables.

Don't expect a fix in Python 3.6 for this, and don't be surprised if Eric/Nick closes this and references whichever bug they're using to track the other work, but it is something that we're aware of and are (slowly) working to address.

OK, Thanks for the quick reply. I have just migrated my application from Python 3.3 to Python 3.6.3 and would not expect you to backport changes from the 3.7 development stream to the released stream. What I will do when I get a chance is pull down the 3.7 sources and compare the demo app I have produced with 3.7 vs 3.6.3 to see if improvements have been done in this area.

I think it's reasonable to keep this open if the problems can be reproduced with the current 3.7 dev branch - while we've refactored a lot of things, we haven't specifically gone memory leak hunting for the embedded case, and I'm pretty sure there are still some items where we delay cleaning them up until the next Py_Initialize call.

The relevant embedding test case is the one at https://github.com/python/cpython/blob/master/Programs/_testembed.c#L41, but I'm not sure we regularly run that in a way that would reliably detect memory leaks (it runs in a subprocess, rather than the main process, so the refleak hunting runs may not be handling it properly).

I don't think the refleak handling would detect leaks across PyInitialize/PyFinalize calls anyway.

Really, we probably just need to loop Initialize/Finalize a few hundred times and measure the memory usage of the process before/after. That will handle a range of leaks, though it doesn't necessarily help us track them down.

Python doesn't clean everything at exit: see bpo-29881.

I just modified Py_Main() in bpo-32030 to clean a few more bytes (like program_name): see pymain_free() in Modules/main.c.

It's basically a duplicate of bpo-1635741.