This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: zlib.compressobj took too much memory on window
Type: resource usage Stage: resolved
Components: Windows Versions: Python 3.7, Python 3.6, Python 3.3, Python 3.4, Python 3.5, Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: dennisding, iritkatriel, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2017-06-29 05:56 by dennisding, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (2)
msg297257 - (view) Author: dennis (dennisding) Date: 2017-06-29 05:56
I'm working on a MMORPG Game server, and using the 32bit window Python(Python2.7.9 and Python3.6.0).
And i create a zlib.compressobj for every connection.

when the connection reach about 7500, the program raise an MemoryError exception. The Program only took me 700MB in memory, But the committed memory is about 2GB.
Here is my test result and test code:
-----------test result------------------
7613 compressobj created!
('memory info:', pmem(rss=736342016, vms=2057256960, num_page_faults=179812, pea
k_wset=736342016, wset=736342016, peak_paged_pool=130760, paged_pool=130584, pea
k_nonpaged_pool=28344, nonpaged_pool=28344, pagefile=2057256960, peak_pagefile=2
057256960, private=2057256960))

--------------test code ----------------
# -*- encoding:utf-8 -*-

import os
import psutil
import zlib

def test():
	result = []
	for i in range(10000):
		try:
			result.append(zlib.compressobj())
		except:
			print('%d compressobj created!'%(i))
			pid = os.getpid()
			p = psutil.Process(pid)
			print('memory info:', p.memory_info())
			break


if __name__ == '__main__':
	test()
msg391452 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-04-20 16:48
It looks like you're not using this API correctly, it doesn't make sense to create 10000 compressobj()s.   You should create one and then call compress() on it as many times as you need.
History
Date User Action Args
2022-04-11 14:58:48adminsetgithub: 74983
2021-04-22 03:35:02methanesetstatus: pending -> closed
stage: resolved
2021-04-20 16:48:08iritkatrielsetstatus: open -> pending

nosy: + iritkatriel
messages: + msg391452

resolution: not a bug
2017-06-29 05:56:39dennisdingcreate