New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tokenize.open() leaks an open binary file on TextIOWrapper error #68028
Comments
If TextIOWrapper constructor fails in tokenize.open(), the open binary file is not closed and a ResourceWarning is emited. Try attached patch: $ python3 -Wd ~/tokenize_open_bug.py
tokenize.open failed: unknown encoding for 'test.py': xxx
/home/haypo/tokenize_open_bug.py:13: ResourceWarning: unclosed file <_io.BufferedReader name='test.py'>
print("tokenize.open failed: %s" % err) The fix is quite simple: add "try: ... except: buffer.close(); raise". If someone wants to fix this issue, an unit test must be added, test based on my script and ensures that the buffer is closed (ex: mock tokenize._builtin_open and checks that close() was called). |
I sent an email to the python core mentorship mailing list to find a volunteer to fix this easy issue. |
ITSM it's not the TextIOWrapper but the detect_encoding fails and throws an error. So I wrapped a try/catch block around that. |
Comparing to introom. Any reason to not wrap the entire in a try? Attached patch. |
Curious, 1227 def test_main(): |
I believe it goes here under the TestTokenize class. Going to give it a shot now. |
Oh, right. But TextIOWrapper can fail for differen reasons. For example, CTRL+c may send KeyboardInterrupt. Try for example: with unittest.mock.patch.object(tokenize, '_builtin_open') as mock_open:
mock_file = mock_open.return_value
mock_file.tell.side_effect = OSError
mock_file.readline.return_value = b''
tokenize.open(fn) This example raises an OSError in TextIOWrapper on file.tell(), and file.close() is not called. |
On Wed, Apr 1, 2015 at 6:18 PM, STINNER Victor <report@bugs.python.org> wrote:
Yes. Any comment on the test? I wil update a second patch. Regards. -- 吾輩は猫である。ホームーページはhttp://introo.me。 |
Yes, as I wrote, I reviewed patches. You may get a notification by email. Anyway, it's here: |
Updated patch based off of haypo's comment. Changed "except Exception as err:" -> "except" |
Based upon the previous review. Catch the TextWrapper, move the test into a function. |
Based off of storchaka's comment, moving the |
tokenize.patch is wrong: you should not call buffer.close() if TextIOWrapper() was created successfully. |
tokenizeV2.patch and tokenize.patch have issues, I reviewed them. Can someone please write a new patch taking my comments in account? |
Hey Haypo, You should see the new file in the next 30 minutes. |
I don't see the new file. |
New changeset 623e07ea43df by Victor Stinner in branch '3.4': New changeset a640d268ba97 by Victor Stinner in branch '3.5': New changeset c5cfd6353b4b by Victor Stinner in branch 'default': |
Sorry for not being more available for feedback on patches. I chose to write the final patch because patches were not updated to take in account my latest comments. I hope that this issue helps you at least the process for reviewing patches ;-) |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: