classification
Title: subprocess _active.remove(self) self not in list _active
Type: Stage:
Components: Library (Lib) Versions: Python 2.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: astrand Nosy List: astrand, atila-cheops, georg.brandl, hvb_tup, sf-robot, tfaujour
Priority: normal Keywords:

Created on 2005-05-10 18:24 by atila-cheops, last changed 2006-05-15 02:20 by sf-robot. This issue is now closed.

Files
File name Uploaded Description Edit
upgrade_2192.py atila-cheops, 2005-09-19 09:31 main script that causes the bug
Messages (16)
msg25262 - (view) Author: cheops (atila-cheops) Date: 2005-05-10 18:24
I start a subprocess in a seperate thread (25 concurrent 
threads)
in some of the threads the following error occurs
 
Exception in thread Thread-4:
Traceback (most recent call last):
  File "C:\Python24\lib\threading.py", line 442, in 
__bootstrap
    self.run()
  File "upgrade.py", line 45, in run
    returncode = p.wait()
  File "C:\Python24\lib\subprocess.py", line 765, in wait
    _active.remove(self)
ValueError: list.remove(x): x not in list
 
this is the code that starts the subprocess and where I 
wait for the result
 
p = subprocess.Popen('command', \
                             stdin=None, 
stdout=subprocess.PIPE, \
                             stderr=subprocess.STDOUT, 
universal_newlines=True)
returncode = p.wait()
errormessage = p.stdout.readlines()
msg25263 - (view) Author: cheops (atila-cheops) Date: 2005-05-12 10:17
Logged In: YES 
user_id=1276121

this might be related to bug 1183780
msg25264 - (view) Author: Peter ├ůstrand (astrand) * (Python committer) Date: 2005-06-23 16:03
Logged In: YES 
user_id=344921

I believe it should be sufficient to add a try...except
clause around _active.remove(). Can you upload a complete
example that triggers the bug? Have you noticed this bug on
Windows, UNIX or both platforms?
msg25265 - (view) Author: cheops (atila-cheops) Date: 2005-09-19 09:29
Logged In: YES 
user_id=1276121

I noticed this bug under windows
to reproduce the bug, I attached the script I use, but this 
needs input, I tried with a simpler example (pinging a number 
of host concurrently) but that did not cause the bug.
msg25266 - (view) Author: HVB bei TUP (hvb_tup) Date: 2006-01-23 16:30
Logged In: YES 
user_id=1434251

I have a similar problem.
Python 2.4.1 under MS Windows 2003,
Multi-Threaded application (about concurrent 10 threads).

In my case the same error occurs during _cleanup called 
from __init__ :

  
File "E:\lisa_ins\ewu\coop\reporting\python\tup_lisa\util\t
hreadutil.py", line 582, in callSubProcess
    creationflags = creationflags
  File "C:\Python24\lib\subprocess.py", line 506, in 
__init__
    _cleanup()
  File "C:\Python24\lib\subprocess.py", line 414, in 
_cleanup
    inst.poll()
  File "C:\Python24\lib\subprocess.py", line 755, in poll
    _active.remove(self)
ValueError: list.remove(x): x not in list

Is there a work-around?
msg25267 - (view) Author: HVB bei TUP (hvb_tup) Date: 2006-01-23 16:34
Logged In: YES 
user_id=1434251

BTW: In my case, I call python.exe from a Windows service.
msg25268 - (view) Author: cheops (atila-cheops) Date: 2006-01-25 07:08
Logged In: YES 
user_id=1276121

As suggested by astrand
adding a try ... except clause in the file subprocess.py did
the job
I had to add that try ... except clause in 2 places
if you look in the file there are 2 instances were
list.remove(x) occurs unprotected.

try:
 list.remove(x)
except:
 pass

I have worked with 2.4.0, 2.4.1 and 2.4.2 and all three
needed the patch.
Hope this helps.
msg25269 - (view) Author: HVB bei TUP (hvb_tup) Date: 2006-01-25 08:10
Logged In: YES 
user_id=1434251

Wouldn't it be best to completely remove any references 
to "_active" and "_cleanup"?
msg25270 - (view) Author: Tristan Faujour (tfaujour) Date: 2006-03-28 23:17
Logged In: YES 
user_id=1488657

I am running into the same problem on a Windows 2k/XP
plateform with a multi-threaded application. It occurs randomly.

It has never appened (yet) on a Linux plateform.

I think accesses to _active should be serialized in a
thread-safe way. _active could be a Queue (from the Queue
module) for example.
msg25271 - (view) Author: Peter ├ůstrand (astrand) * (Python committer) Date: 2006-03-29 05:11
Logged In: YES 
user_id=344921

>I think accesses to _active should be serialized in a
>thread-safe way. _active could be a Queue (from the Queue
>module) for example

Simply list operations such as remove() and append() are
thread safe, so there should be no need for a Queue. Also, a
Queue cannot be used since the subprocess module needs to be
compatible with Python 2.2. 
msg25272 - (view) Author: Tristan Faujour (tfaujour) Date: 2006-03-29 13:50
Logged In: YES 
user_id=1488657

> Simply list operations such as remove() and append() are
> thread safe,

OK, my apologies... I did not know.

I did some more tests. On Linux, I found lots of:

  File "./subprocess.py", line 428, in call
    return Popen(*args, **kwargs).wait()
  File "./subprocess.py", line 1023, in wait
    pid, sts = os.waitpid(self.pid, 0)
OSError: [Errno 10] No child processes

The try...except solution does not handle this (since we are
in the "posix" section).

In my opinion, the call to _cleanup() in Popen.__init__() is
useless (it just checks if older processes have stopped when
a new one is started. I cannot see why it could be
mandatory) and it is the root of this bug.

I commented it out (line 506) and retried my tests on Linux
& Windows plateforms: I had no exception at all.

msg25273 - (view) Author: cheops (atila-cheops) Date: 2006-04-10 14:57
Logged In: YES 
user_id=1276121

see patch #1467770
msg25274 - (view) Author: HVB bei TUP (hvb_tup) Date: 2006-04-11 07:21
Logged In: YES 
user_id=1434251

I looked at the patch #1467770 you mentioned and 
downloaded the patch.

Is it correct that _deadlock variable is used in the 
definition of poll but it is not in the argument list?
msg25275 - (view) Author: cheops (atila-cheops) Date: 2006-04-11 20:10
Logged In: YES 
user_id=1276121

there are 2 definitions of the function poll, one for 
windows and one for POSIX systems
in the windows version _deadlock is not used
in the POSIX version _deadlock is used

see also modification made by loewis in committed version 
45234
msg25276 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2006-04-30 19:52
Logged In: YES 
user_id=849994

The error causing "_active.remove(self)" call was removed
from wait() with patch 1467770, so I think this is fixed.
msg25277 - (view) Author: SourceForge Robot (sf-robot) Date: 2006-05-15 02:20
Logged In: YES 
user_id=1312539

This Tracker item was closed automatically by the system. It was
previously set to a Pending status, and the original submitter
did not respond within 14 days (the time period specified by
the administrator of this Tracker).
History
Date User Action Args
2005-05-10 18:24:30atila-cheopscreate