This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author Александр Карпинский
Recipients Александр Карпинский
Date 2016-07-17.01:53:40
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1468720423.06.0.361257976034.issue27535@psf.upfronthosting.co.za>
In-reply-to
Content
Actually, this issue is related to the warning module. The test script creates a lot of files with different names and deletes them. On the first pass the scripts calls the `f.close()` method for every file. On the second it doesn't.

As a result, on the first pass, the memory consumption is constant and is about 3.9 Mb for my environment. For the second pass, the memory consumption constantly grows up to 246 Mb for 1 million files. I.e. memory leak is about 254 bytes for every opened file.

This happens because of warning about not closed files. The memory is consumed for storing all possible warning messages generated for different file names to prevent further warnings with the same messages. Which is not the case if we are always opening new files.

While not calling f.close() **might** lead to memory issues, the warning which notifies about this **absolutely guaranteed** leads to memory issue. Although f.close() is highly recommended, this is still the issue for code which doesn't use it for some reasons and doesn't experience other problems with not closed files because files are actually always closed in IOBase.__del__.

This issue was discovered in this report: https://github.com/python-pillow/Pillow/issues/2019


The best way to fix this is excluding file name from warning message. As a result, the message for every file will be the same and warnings registry will not grow. For now, the message looks like this:

./memory.py:23: ResourceWarning: unclosed file <_io.FileIO name='__999.tmp' mode='wb' closefd=True>
  open_files(number, close)

It may looks like this:

./memory.py:23: ResourceWarning: unclosed file <FileIO mode='wb' closefd=True>
  open_files(number, close)
History
Date User Action Args
2016-07-17 01:53:43Александр Карпинскийsetrecipients: + Александр Карпинский
2016-07-17 01:53:43Александр Карпинскийsetmessageid: <1468720423.06.0.361257976034.issue27535@psf.upfronthosting.co.za>
2016-07-17 01:53:42Александр Карпинскийlinkissue27535 messages
2016-07-17 01:53:41Александр Карпинскийcreate