This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author Fry-kun
Recipients Fry-kun, amaury.forgeotdarc, georg.brandl, hunteke, pitrou
Date 2010-10-27.20:06:31
SpamBayes Score 9.37664e-08
Marked as misclassified No
Message-id <1288209993.26.0.810003595522.issue9942@psf.upfronthosting.co.za>
In-reply-to
Content
This issue sounds very interesting to me for a somewhat different reason.
My problem is that I'm trying to run multiple processes on separate CPUs/cores with os.fork(). In short, the data set is the same (~2GB) and the separate processes do whatever they need, although each fork treats the data set as read-only.
Right after the fork, data is shared and fits in RAM nicely, but after a few minutes each child process runs over a bunch of the data set (thereby modifying the ref counters) and the data is copied for each process. RAM usage jumps from 15GB to 30GB and the advantage of a fork is gone.

It would be great if there was an option to separate out the ref counters for specific data structures, since it's obviously a bad idea to turn it on by default for everything and everyone.
History
Date User Action Args
2010-10-27 20:06:33Fry-kunsetrecipients: + Fry-kun, georg.brandl, amaury.forgeotdarc, pitrou, hunteke
2010-10-27 20:06:33Fry-kunsetmessageid: <1288209993.26.0.810003595522.issue9942@psf.upfronthosting.co.za>
2010-10-27 20:06:31Fry-kunlinkissue9942 messages
2010-10-27 20:06:31Fry-kuncreate