Message334641
interestingly, while it is expected Process or Queue would actually close resource file descriptors and doesn't because a dev decided they prefer to defer to the user how to manage gc themselves, the interesting thing is if you 'upgrade' your code to use a pool, the process fd will be closed as the pool will destroy the object (so it is gc more often);
Say you're limited to a little over 1000 fd in your o/s you can do this
#######################################################################
import multiprocessing
import json
def process(data):
with open('/tmp/fd/%d.json' % data['name'], 'w') as f:
f.write(json.dumps(data))
return 'processed %d' % data['name']
if __name__ == '__main__':
pool = multiprocessing.Pool(1000)
try:
for _ in range(10000000):
x = {'name': _}
pool.apply(process, args=(x,))
finally:
pool.close()
del pool
#######################################################################
only the pool fd hangs around longer then it should, which is a huge improvement, and you might not find a scenario where you need many pool objects. |
|
Date |
User |
Action |
Args |
2019-02-01 01:30:01 | Chris Langton | set | recipients:
+ Chris Langton, pitrou, davin, Henrique Andrade, pablogsal |
2019-02-01 01:29:59 | Chris Langton | set | messageid: <1548984599.6.0.785921805558.issue33081@roundup.psfhosted.org> |
2019-02-01 01:29:59 | Chris Langton | link | issue33081 messages |
2019-02-01 01:29:59 | Chris Langton | create | |
|