This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: subprocess leaks open file descriptors between Popen instances causing hangs
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.2, Python 2.7
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: Giovanni.Bajo, georg.brandl, gregory.p.smith, jwilk, milko.krachounov, ned.deily, paul.moore, vstinner
Priority: normal Keywords: patch

Created on 2009-10-26 23:14 by milko.krachounov, last changed 2022-04-11 14:56 by admin. This issue is now closed.

File name Uploaded Description Edit
subprocess-cloexec-py3k.patch milko.krachounov, 2010-12-10 19:44 [obsolete] Simple patch to enable CLOEXEC on subprocess pipes
subprocess-00-subprocess_test_utils.patch milko.krachounov, 2010-12-11 22:26 Utils required by the tests of the CLOEXEC patches (issue 6559 could also use them for tests)
subprocess-01-atomic_cloexec_pipe2.patch milko.krachounov, 2010-12-11 22:26 Create subprocess pipes with FD_CLOEXEC (atomically when possible)
subprocess-02-cloexec_tests.patch milko.krachounov, 2010-12-11 22:33 Tests for FD_CLOEXEC on the subprocess pipes
Messages (28)
msg94538 - (view) Author: Milko Krachounov (milko.krachounov) Date: 2009-10-26 23:14
Currently, close_fds defaults to False. The are few cases in which one
would want to leave the fds open, in all the rest leaving them open can
lead to unpleasant side effects. For example, the following doesn't work:

>>> p1 = Popen(['cat'], stdin=PIPE, stdout=PIPE)
>>> p2 = Popen(['grep', 'a'], stdin=p1.stdout, stdout=PIPE)
>>> p1.stdin.write("aaaaaaaaaaaaaaaa\n")
>>> p1.stdin.close()

It would block forever, and it is not obvious that p1.stdin remains
open, because p2 was created without close_fds=True. On the other hand,
in each and every case where close_fds=True is required, the programmer
is aware that he needs to leave some fds open (and usually knows which fds).

The current default is harmful, because in each case where the file
descriptors are not explicitly needed in the child process, they can
cause problems hard to debug. It seems that only about 10% of the
current uses of Popen have close_fds=True.

I propose that the close_fds default is changed to True. Alternatively,
this could be combined with bug #6559 by completely removing the
close_fds argument, and leaving only pass_fds, which could accept
subprocess.ALL_FDS as a value.

There are two issues with my proposal:
1. close_fds would have to be changed to give a warning each time it is
not specified before the change of the default can be adopted. Otherwise
it would break any programs relying on close_fds being False by default.
2. Closing fds has a slight performance impact.

However, I think that close_fds=True is much more sensible default. I
think it will be a good idea if at least py3k adopts the change.
msg100372 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2010-03-04 03:08
issue 2320 is somewhat related to this.
msg123408 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2010-12-04 23:14
This bug appears to be Unix-only. On Windows:

>>> from subprocess import *
>>> p1 = Popen(['cat'], stdin=PIPE, stdout=PIPE)
>>> p2 = Popen(['grep', 'a'], stdin=p1.stdout, stdout=PIPE)
>>> p1.stdin.write("aaaaaaaaaaaaaaaa\n")
>>> p1.stdin.close()

So there's no clear reason why the default should change on Windows. (It's not possible to specify close_fds explicitly on Windows for this case:

>>> p1 = Popen(['cat'], stdin=PIPE, stdout=PIPE, close_fds=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\Apps\Python27\lib\", line 630, in __init__
    raise ValueError("close_fds is not supported on Windows "
ValueError: close_fds is not supported on Windows platforms if you redirect stdin/stdout/stderr

which may imply that False is not only reasonable, but necessary on Windows. I haven't dug into this enough to know for sure if this is the case, though).
msg123737 - (view) Author: Giovanni Bajo (Giovanni.Bajo) Date: 2010-12-10 16:46
Hi Gregory, I saw your commit here:

This basically means that in 3.2 it is mandatory to specify close_fds to avoid a DeprecationWarning. *BUT* there is no good value that works both on Windows and Linux if you redirect stdout/stderr, as shown in this bug.

So basically in 3.2 to avoid a warning, each and every usage of Popen() with a redirection should be guarded by an if that checks the platform. I don't think this is acceptable.

Have I misunderstood something? Also: can you please explain how the behaviour is going to change in 3.3? I assume that you are planning to change the default to True; but would that also cover Windows' singularity in redirection cases?
msg123741 - (view) Author: Milko Krachounov (milko.krachounov) Date: 2010-12-10 18:46
I'd offer two ideas.

1. Add a constant DISREGARD_FDS to the subprocess module could help. It would allow the user to specify his intent, and let the implementation choose the best action. Popen(..., close_fds=subprocess.DISREGARD_FDS) would mean that any fds additional fds are not needed, and can be closed if it is determined that it is the best course of action.

So, if DISREGARD_FDS gets passed on Windows assume close_fds=False, while on POSIX assume close_fds=True (although this has its downsides, see at the bottom).
2. Set the CLOEXEC flag on the other side of the pipes after the fork. With the attached patch, I don't get the issue I reported:

Python 3.2b1 (py3k:87158, Dec 10 2010, 20:13:57) 
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from subprocess import *
>>> p1 = Popen(['cat'], stdin=PIPE, stdout=PIPE, close_fds=False)
>>> p2 = Popen(['grep', 'a'], stdin=p1.stdout, stdout=PIPE, close_fds=False)
>>> p1.stdin.write(b"aaaaaaaaaaaaaaaa\n")
>>> p1.stdin.close()

Additional problem with close_fds=True is that if MAXFD is huge, it can cause a huge problem with performance, so perhaps the default being True isn't all that great. On Linux you can close only the open fds, but I'm not sure if that's possible on other POSIX systems.
msg123742 - (view) Author: Milko Krachounov (milko.krachounov) Date: 2010-12-10 19:15
The cloexec approach still doesn't help with issue 2320. In fact, with threading and people calling subprocess from multiple threads, *this* issue wouldn't be fixed with my patch either unless mutexes are used. It's impossible to avoid a race here with threads and no mutex as far as I can tell. It would mean that the creation of the pipe, the fork and the setting of the CLOEXEC flag needs to be done atomically... yikes.
msg123743 - (view) Author: Giovanni Bajo (Giovanni.Bajo) Date: 2010-12-10 19:30
Setting CLOEXEC on the pipes seems like a very good fix for this bug. I'm +1 on it, but I think it should be the default; instead, your proposed patch adds a new argument to the public API. Why do you think it's necessary to do so?

At the same time, we need a solution to handle close_fds, because the current status of the 3.2 with the DeprecationWarning (on 90% of subprocess uses in the world, if it ever gets backported to 2.7) and no way to fix it in a multi-platform way is really sad.

I don't think a new constant DISREGARD_FDS is necessary. I think we can just use "None" as intermediate default (just like the current 3.2 does), and later switch it to True. The only further required action is to make "True" always work on Windows and never error out (just make it do nothing if there are some redirections), which is an obviously good thing to do to increase portability of subprocess.

Otherwise, if this can't make 3.2, I think the DeprecationWarning should be reverted until we agree on a different solution.
msg123744 - (view) Author: Milko Krachounov (milko.krachounov) Date: 2010-12-10 19:44
> I'm +1 on it, but I think it should be the default; instead, 
> your proposed patch adds a new argument to the public API. Why do you 
> think it's necessary to do so?

I don't think it's necessary. I put it there because when I was testing I thought it might help. For example, you might want to keep the pipes open and then open another process with close_fds=False, thus changing the current default might cause some regressions in some software and an argument would allow an easier transition. I updated the patch removing anything unnecessary. Though it still has a huge race when used with threading.
msg123746 - (view) Author: Giovanni Bajo (Giovanni.Bajo) Date: 2010-12-10 20:00
Would you mind elaborating on where is the race condition?
msg123748 - (view) Author: Milko Krachounov (milko.krachounov) Date: 2010-12-10 20:10
It's almost exactly the same race condition as the one described in issue 2320. The pipes are created and stay without the CLOEXEC flag for a while (until the process has been forked and fcntl has been called). During that time another thread can launch a subprocess, the subprocess will inherit the pipe, and there might be a hang similar to the one described here (or in issue 2320).
msg123754 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-12-10 22:02
(Adding the 3.2 release manager: a potential release blocker?)
msg123763 - (view) Author: Milko Krachounov (milko.krachounov) Date: 2010-12-11 01:32
I created another patch that attempts to create the pipes atomically.

On GNU/Linux, if pipe2 is available, it uses it to create the pipes, and there is no race. On other POSIX platforms, pipe and fcntl are called without releasing the GIL - relying on the fact that _posixsubprocess.fork_exec doesn't release the GIL either, so the two can't run at the same time (bonus: os.fork doesn't release the GIL either). I can't reproduce neither issue 7213 nor issue 2320 with either implementation, so the patch seems to fix them.

1. If the _posixsubprocess module isn't compiled, the race still exists (well, without it subprocess isn't safe to use with threads anyway).
2. On GNU/Linux systems where glibc was compiled with kernel headers >=2.6.27, but the running kernel is <2.6.27, the subprocess module won't work. (There should be a fix for that?)
3. I have no way to tell that the non-Linux implementation works for sure. I've been running it in an endless loop, and so far there have been no hangs (*), but that doesn't mean that it doesn't have a rare race that's beyond my comprehension. With pipe2() you can be certain, but I have my doubts about the other implementation.

All unit tests seem to pass.

(*) Actually, I *thought* it hang on my first attempt, but I interrupted the process too soon to tell for sure. No hangs after that. :(
msg123766 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-12-11 02:20
Can you add a test to your patch?
msg123788 - (view) Author: Milko Krachounov (milko.krachounov) Date: 2010-12-11 12:34
I attached unit tests that test that cloexec is properly set. I can't test my tests too well with the unpatched version because is too complicated to use, and doesn't print any useful output by default.
msg123792 - (view) Author: Milko Krachounov (milko.krachounov) Date: 2010-12-11 14:12
I add a patch that tests close_fds (there's no test for close_fds), that requires the tests1 patch. By the way, should there be a test for the atomicity of the operations?
msg123801 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-12-11 16:55
subprocess-cloexec-atomic-py3k-tests2-close_fds.patch adds a test called to Win32ProcessTestCase which is specific to Windows. And this class has already a test with the same name. You should move your test to ProcessTestCase (and so it will test the C implementation, the Python implementation and also without poll).
msg123802 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-12-11 17:00
> subprocess-cloexec-atomic-py3k-tests2-close_fds.patch adds a test
> called [test_close_fds] to Win32ProcessTestCase ...

Oops, forget my last comment, I didn't applied the patches in the right order. There are too much patches :-p Can you try to create one unique patch? It will be easier to test it and to review it.
msg123804 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-12-11 17:02

+case $ac_sys_system in
+  GNU*|Linux*)
+      AC_CHECK_FUNC(pipe2, AC_DEFINE(HAVE_PIPE2, 1, [Define if the OS supports pipe2()]), )
I think that you can remove the test on the OS name. AC_CHECK_FUNC() doesn't hurt if the function doesn't exist. Other OS may have pipe2().
msg123806 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-12-11 17:12
test_pipe_cloexec_unix_tools() is specific to UNIX/BSD because it requires cat and grep programs. You should try to reuse the Python interpreter to have a portable test (eg. working on Windows), as you did with

+        data = b'aaaaaaaaaaaaaaaaaaaa\n'
+        subdata = b'aaa'
+        assert subdata in data, "Test broken"

Use maybe subdata = data[:3] to avoid an assertion.

I don't understand why do you talk about "atomicity". Do you test add non-atomic operations? Was subprocess atomic?

If I understood correctly, you are fixing a specific issue which can be called something like "subprocess: close pipes on exec(), set FD_CLOEXEC flag to all pipes", and no more changing the default value of close_fds. Can you update the title please?
msg123817 - (view) Author: Milko Krachounov (milko.krachounov) Date: 2010-12-11 22:26
OK, I have created new updated patches. I haven't combined them in one patch because some of the changes can be applied independently, the three patches can be cat'ed together if anyone sees separate patches a problem. ;)

I. Changes:
* Now pipe2 would be used on all systems where available.
* I don't depend on external grep and cat, but use utils shipped with the tests.

II. The patches:
1. subprocess-00-subprocess_test_utils.patch
This contains the and utilities that are needed by the tests. A separate file because they can be used for creating unit tests for issue 6559. In case these patches aren't accepted, it could be useful there.

2. subprocess-01-atomic_cloexec_pipe2.patch
This file contains the changes that make the subprocess pipes created with the FD_CLOEXEC flag (atomically through pipe2) where possible. Can be used independently of the rest
of the patches.

3. subprocess-02-cloexec_tests.patch
This file contains the tests that verify that CLOEXEC is set on the pipes created by subprocess and this allows complex pipes between apps to work correctly. Requires subprocess-00-subprocess_test_utils.patch. It can be used for another implementation of CLOEXEC in case this implementation isn't accepted. With slight modification it can be used to test that pipes work fine with close_fds=True and no CLOEXEC.

III. Atomicity and issue 2320:

> I don't understand why do you talk about "atomicity". Do you test add non-atomic 
> operations? Was subprocess atomic?
> If I understood correctly, you are fixing a specific issue which can be called 
> something like "subprocess: close pipes on exec(), set FD_CLOEXEC flag to all pipes", 
> and no more changing the default value of close_fds. Can you update the title please?

The CLOEXEC flag needs to be set atomically (or at least in a way that another subprocess won't start in the middle of it) so that issue 2320 is also resolved. This is why I suggested the use of pipe2. On systems where pipe2 isn't available, the patches rely on the coincidence that the GIL during both process execution and the pipe creation. I was wondering if a tests that verify for things like issue 2320 are wanted in general (there's no way to reliably test such, but there could be a test that does it with a certain probability).

In short, yes, the suggested patches attempt to solve this issue without a change to the default, but also to resolve issue 2320 (changing the default would also resolve it).

III. More issues:

1. What other possible situations exist causing the same issue that can be solved with close_fds=True and are caused by resources coming from other modules? Are any of those common? Maybe something involving os.openpty() and subprocess? Does anyone use those together? Are such issues worth worrying about (in other words, is changing the default still necessary if the subprocess pipes have the FD_CLOEXEC flag?)
2. What possible side-effects could the FD_CLOEXEC flag have? Any common use-case that is broken by the fact that the pipes are now with FD_CLOEXEC?
3. Can any changes be done to Python 2.7's subprocess? Can the issue be fixed there too, or it will have to remain as it is now?
msg123818 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2010-12-11 23:07
Paul & Giovanni: yes I hadn't given the windows side of things any thought when I made the change for beta1.

Milko: The DISREGARD_FDS approach is basically what I was intending to do.

Also, there really wasn't any objection to going ahead and changing the close_fds default for real in 3.2 assuming the windows issue is addressed.  changing it to your disregard suggestion (true on posix, false on windows) seems to fit peoples needs and keeps the basic usage simple and gets rid of any suggestion that everyone needs to specify true or false for all use cases.
msg123832 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-12-12 10:11

+    _MAXFD = os.sysconf("SC_OPEN_MAX")
+    _MAXFD = 256

It looks like this code (256 constant) comes from Is that a good value? On Linux, SC_OPEN_MAX is usually 1024, and it can be 4096. Should we keep the default value 256, or use 1024 or 4096 instead? I don't know on which OS SC_OPEN_MAX is missing.

-- isopen(fd) uses fcntl.fcntl(fd, fcntl.F_GETFD, 0). Is it always available? If not, we can add fstat() as a fallback. The buildbots will tell us :-)

-- _create_pipe() doesn't use pipe2() because Python doesn't provide pipe2(). We should maybe add it to the posix module (open maybe a new issue for that).

> The CLOEXEC flag needs to be set atomically (or at least in a way
> that another subprocess won't start in the middle of it)

For the Python implementation, the GIL is not enough to ensure the atomicity of a process creation. That's why _posixsubprocess was created. I suppose that other parts of subprocess are not atomic and a lock is required to ensure that the creation of subprocess is atomic.


_posixsubprocess.c: is FD_CLOEXEC flag always available? fcntlmodule.c uses a "#ifdef FD_CLOEXEC".

Can you add a comment to explain why you can release the GIL here? (because the operation is atomic)

+    res = pipe2(fds, O_CLOEXEC);

-- test_pipe_cloexec() and test_pipe_cloexec_real_tools() should maybe be skipped if fcntl has no attribute FD_CLOEXEC.
msg123837 - (view) Author: Milko Krachounov (milko.krachounov) Date: 2010-12-12 13:14
> For the Python implementation, the GIL is not enough to
> ensure the atomicity of a process creation. That's why 
> _posixsubprocess was created. I suppose that other parts 
> of subprocess are not atomic and a lock is required to
> ensure that the creation of subprocess is atomic.
Both _posixsubprocess.fork_process and _posixsubprocess.cloexec_pipe (without HAVE_PIPE2) hold the GIL. They can't be running at the same time, so in regards to each other, they will be atomic. Or at least that's the idea. You don't need a separate lock when the GIL is already held. Neither a lock nor the GIL make the operation to be atomic (e.g. if someone forks from multiprocessing, the effect might be different, though os.fork() also holds the GIL and I think multiprocessing uses it).

The idea of _posixsubprocess is that fork() isn't thread-safe, because Python might call malloc() or free() in the child, which might be currently locked by another thread, which might cause the child to hang. Nothing to do with atomicity.

> Can you add a comment to explain why you can release the 
> GIL here? (because the operation is atomic)

I release the GIL because I copy-pasted the code from os.pipe(). It makes sense that pipe() and pipe2() are called in the same way. In the other implementation it is held to (ab)use it to avoid the race, but there's no need to hold it here because pipe2() *is* atomic.
msg123863 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2010-12-13 06:45
The close_fds default has been fixed in r87206 to remove the DeprecationWarning and remain False on Windows.  It changes to True on POSIX.
msg123864 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2010-12-13 08:03
Milko's subprocess-00/01/02 patch set have been committed with minor modifications in r87207 & r87208.  Thanks, especially for the test cases!

Is there anything else left that we know about for this bug?
msg123869 - (view) Author: Giovanni Bajo (Giovanni.Bajo) Date: 2010-12-13 10:02
Hi Gregory,

will you backport Mirko's patches to subprocess32?

The last thing left in this bug is my proposal to change the default of close_fds to True to Windows too, but at the same time detect whether this is possible or not (depending on the pipe redirections). 

So basically close_fds=True would be changed to mean "close the FDs, if it is possible, otherwise never mind". This is not a break in compatibility on Linux/UNIX (where it is always "possible"), nor Windows (where currently it just raises a ValueError if you ask it to raise close the file descriptors while doing redirections).

The rationale for this change is again cross-compatibility. I don't like when my code breaks because of a limitation of an OS that has a clear workaround. Subprocess is a high-level library after all, it's not like os.fork() or similar low-level libraries which expose the underlying platform differences.
msg123940 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2010-12-14 13:47
I updated the documentation and changed the close_fds default on Windows to be True when possible per Giovanni's suggestion in r87229.  That keeps the API and defaults as consistent as possible across all platforms.
msg123941 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2010-12-14 13:48
P.S.  Yes I will be backporting all of this to subprocess32.
Date User Action Args
2022-04-11 14:56:54adminsetgithub: 51462
2010-12-14 13:48:51gregory.p.smithsetmessages: + msg123941
2010-12-14 13:47:59gregory.p.smithsetstatus: open -> closed
resolution: accepted
messages: + msg123940
2010-12-13 10:02:49Giovanni.Bajosetmessages: + msg123869
2010-12-13 08:03:14gregory.p.smithsetmessages: + msg123864
2010-12-13 06:45:59gregory.p.smithsetmessages: + msg123863
2010-12-12 13:14:49milko.krachounovsetmessages: + msg123837
2010-12-12 10:11:12vstinnersetmessages: + msg123832
2010-12-11 23:07:47gregory.p.smithsetmessages: + msg123818
2010-12-11 22:34:53milko.krachounovsetfiles: - subprocess-cloexec-atomic-py3k-tests2-close_fds.patch
2010-12-11 22:34:48milko.krachounovsetfiles: - subprocess-cloexec-atomic-py3k-tests1.patch
2010-12-11 22:34:43milko.krachounovsetfiles: - subprocess-cloexec-atomic-py3k.patch
2010-12-11 22:33:06milko.krachounovsetfiles: + subprocess-02-cloexec_tests.patch
2010-12-11 22:32:53milko.krachounovsetfiles: - subprocess-02-cloexec_tests.patch
2010-12-11 22:27:32milko.krachounovsetfiles: + subprocess-02-cloexec_tests.patch
2010-12-11 22:26:42milko.krachounovsetfiles: + subprocess-01-atomic_cloexec_pipe2.patch
2010-12-11 22:26:07milko.krachounovsetfiles: + subprocess-00-subprocess_test_utils.patch

messages: + msg123817
title: Popen.subprocess change close_fds default to True -> subprocess leaks open file descriptors between Popen instances causing hangs
2010-12-11 18:40:49jwilksetnosy: + jwilk
2010-12-11 17:12:27vstinnersetmessages: + msg123806
2010-12-11 17:02:56vstinnersetmessages: + msg123804
2010-12-11 17:00:18vstinnersetmessages: + msg123802
2010-12-11 16:55:01vstinnersetmessages: + msg123801
2010-12-11 14:12:05milko.krachounovsetfiles: + subprocess-cloexec-atomic-py3k-tests2-close_fds.patch

messages: + msg123792
2010-12-11 12:52:18milko.krachounovsetfiles: + subprocess-cloexec-atomic-py3k-tests1.patch
2010-12-11 12:52:08milko.krachounovsetfiles: - subprocess-cloexec-atomic-py3k-tests1.patch
2010-12-11 12:46:13milko.krachounovsetfiles: + subprocess-cloexec-atomic-py3k-tests1.patch
2010-12-11 12:45:59milko.krachounovsetfiles: - subprocess-cloexec-atomic-py3k-tests1.patch
2010-12-11 12:34:57milko.krachounovsetfiles: + subprocess-cloexec-atomic-py3k-tests1.patch

messages: + msg123788
2010-12-11 02:20:57vstinnersetnosy: + vstinner
messages: + msg123766
2010-12-11 01:32:59milko.krachounovsetfiles: + subprocess-cloexec-atomic-py3k.patch

messages: + msg123763
2010-12-10 22:02:05ned.deilysetnosy: + ned.deily, georg.brandl
messages: + msg123754
2010-12-10 20:10:20milko.krachounovsetmessages: + msg123748
2010-12-10 20:00:14Giovanni.Bajosetmessages: + msg123746
2010-12-10 19:44:28milko.krachounovsetfiles: + subprocess-cloexec-py3k.patch

messages: + msg123744
2010-12-10 19:44:05milko.krachounovsetfiles: - subprocess-cloexec-py3k.patch
2010-12-10 19:30:34Giovanni.Bajosetmessages: + msg123743
2010-12-10 19:15:13milko.krachounovsetmessages: + msg123742
2010-12-10 18:47:00milko.krachounovsetfiles: + subprocess-cloexec-py3k.patch
keywords: + patch
messages: + msg123741
2010-12-10 16:46:13Giovanni.Bajosetnosy: + Giovanni.Bajo
messages: + msg123737
2010-12-04 23:14:57paul.mooresetnosy: + paul.moore
messages: + msg123408
2010-03-04 03:08:10gregory.p.smithsetassignee: gregory.p.smith

messages: + msg100372
nosy: + gregory.p.smith
2010-01-11 21:03:42floxlinkissue7448 superseder
2009-10-26 23:14:01milko.krachounovcreate