Author mythsmith
Recipients mythsmith
Date 2014-02-17.15:47:03
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1392652025.76.0.144236203024.issue20660@psf.upfronthosting.co.za>
In-reply-to
Content
I seems that upon the start of a second manager, all objects referenced in the first one gets an INCREF. On the third start, all objects created by the first and the second manager get another INCREF. And so on. I cannot understand why the start of a totally new manager, in a new process, will cause all these INCREF about object hosted on other managers - which cause the program to take forever to start/stop.

This small script fully reproduces the behaviour, tested in python 2.7 and 3.2:

from __future__ import print_function
import multiprocessing, logging
# # Activate multiprocessing logging
mplog = multiprocessing.get_logger()
mplog.setLevel(multiprocessing.util.DEBUG)
mplog.addHandler(logging.StreamHandler())
objs=[]
def newman(n=50):
    global objs
    m=multiprocessing.Manager()
    print('created')
    for i in range(n):
        objs.append(m.Value('i',i)) 
    return m    

print('#### first man')
m1=newman()

print ('#### second man')
m2=newman()

print ('#### third man')
m3=newman(0)

(Output is attached)

After the start of the first manager, the logger prints out the messages relative to the creation of the first 50 objects.

But when the second manager is starting - before any object was created by it - the logger prints out exactly 50 INCREF messages.Then follows the messages relating to the creation of the 50 new objects on manager 2.

When the third manager starts - before any object was created by it - 100 more INCREF messages are printed.

No object creation message is seen after m3 creation, as I passed 0 to the newman() function.

When the program ends, a similar amount of DECREF messages is printed.

It seems that, when I start a new manager, it creates a reference to all objects referenced by previous managers. In a big application this translates into extremely slow startup/shutdown.
History
Date User Action Args
2014-02-17 15:47:05mythsmithsetrecipients: + mythsmith
2014-02-17 15:47:05mythsmithsetmessageid: <1392652025.76.0.144236203024.issue20660@psf.upfronthosting.co.za>
2014-02-17 15:47:05mythsmithlinkissue20660 messages
2014-02-17 15:47:04mythsmithcreate