Title: Please replace the use of pickle in subprocess with json.
Type: Stage: resolved
Components: Versions:
Status: closed Resolution: duplicate
Dependencies: Superseder: Please replace the use of pickle in multiprocessing with json.
View: 11358
Assigned To: Nosy List: asdfasdfasdfasdfasdfasdfasdf, vstinner
Priority: normal Keywords:

Created on 2011-03-01 07:02 by asdfasdfasdfasdfasdfasdfasdf, last changed 2011-03-01 14:41 by asdfasdfasdfasdfasdfasdfasdf. This issue is now closed.

Messages (8)
msg129744 - (view) Author: david (asdfasdfasdfasdfasdfasdfasdf) Date: 2011-03-01 07:02
Please replace the use of pickle in subprocess with json.
msg129764 - (view) Author: david (asdfasdfasdfasdfasdfasdfasdf) Date: 2011-03-01 13:35
Um this isn't a duplicate this is addressing a different module to multiprocessing. Currently in subprocess you can almost remove the use of pickle with little to no side-effects.
msg129765 - (view) Author: david (asdfasdfasdfasdfasdfasdfasdf) Date: 2011-03-01 13:36
Can you please re-open this bug ? (unless you feel otherwise).
msg129771 - (view) Author: david (asdfasdfasdfasdfasdfasdfasdf) Date: 2011-03-01 13:53
I don't have 3.3 installed so I cannot test it, but here is a patch for 2.6. I am sure it breaks stuff - are there tests for the subprocess module that would cover the cases that pickle was used for? 

---	2011-03-02 00:47:59.000000000 +1100
+++	2011-03-02 00:51:27.000000000 +1100
@@ -414,7 +414,7 @@
     import select
     import errno
     import fcntl
-    import pickle
+    import json
 __all__ = ["Popen", "PIPE", "STDOUT", "call", "check_call", "CalledProcessError"]
@@ -1105,7 +1105,7 @@
                             exc_value.child_traceback = ''.join(exc_lines)
-                            os.write(errpipe_write, pickle.dumps(exc_value))
+                            os.write(errpipe_write, json.dumps(exc_value))
                         # This exitcode won't be reported to applications, so it
                         # really doesn't matter what we return.
@@ -1134,7 +1134,7 @@
             if data != "":
                 _eintr_retry_call(os.waitpid,, 0)
-                child_exception = pickle.loads(data)
+                child_exception = json.loads(data)
                 for fd in (p2cwrite, c2pread, errread):
                     if fd is not None:
msg129772 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-03-01 14:01
Why do you want to replace pickle by json?
msg129776 - (view) Author: david (asdfasdfasdfasdfasdfasdfasdf) Date: 2011-03-01 14:31
From my reading of the code it may be possible if I execute a command via Popen that the child had output that went to stderror, because stderror is associated with the fd of errpipe_write, and it was not to be 'trusted' (lets say I ran it as another user) then it could be pickle.loaded in the parent - and this could potentially be bad.

I could be totally wrong about this tho. I haven't tested the above case yet. 

Regardless - the use of pickle here is not really required and json can do what pickle is doing (from my reading of the code thus far).
msg129777 - (view) Author: david (asdfasdfasdfasdfasdfasdfasdf) Date: 2011-03-01 14:33
Actually I don't think that is possible mmm.
msg129778 - (view) Author: david (asdfasdfasdfasdfasdfasdfasdf) Date: 2011-03-01 14:41
As the child will have already have exec'ed there will be no exception raised --> so the parent shouldn't pickle.load from stderror... So unless there is a path where the parent will end up pickle.load ing the exception that case I put before is not possible.
Date User Action Args
2011-03-01 14:41:29asdfasdfasdfasdfasdfasdfasdfsetmessages: + msg129778
2011-03-01 14:33:03asdfasdfasdfasdfasdfasdfasdfsetmessages: + msg129777
2011-03-01 14:31:33asdfasdfasdfasdfasdfasdfasdfsetmessages: + msg129776
2011-03-01 14:01:27vstinnersetnosy: + vstinner
messages: + msg129772
2011-03-01 13:53:18asdfasdfasdfasdfasdfasdfasdfsetmessages: + msg129771
2011-03-01 13:36:15asdfasdfasdfasdfasdfasdfasdfsetmessages: + msg129765
2011-03-01 13:35:03asdfasdfasdfasdfasdfasdfasdfsetmessages: + msg129764
2011-03-01 07:53:32ezio.melottisetstatus: open -> closed
superseder: Please replace the use of pickle in multiprocessing with json.
resolution: duplicate
stage: resolved
2011-03-01 07:02:24asdfasdfasdfasdfasdfasdfasdfcreate