classification
Title: SocketHandler sends obejcts while they cannot be unpickled on receiver's side
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Nathan Jensen, Zbigniew.Kacprzak, python-dev, vinay.sajip
Priority: normal Keywords:

Created on 2012-03-29 10:03 by Zbigniew.Kacprzak, last changed 2015-03-11 21:00 by Nathan Jensen. This issue is now closed.

Messages (3)
msg157019 - (view) Author: Zbigniew Kacprzak (Zbigniew.Kacprzak) Date: 2012-03-29 10:03
I decided to use SocketHandler in multi-processes application.
Log server, sending data, logging simple strings - works fine.

The problem is with own classes (or external libraries).
Looks like SocketHandler creates pickles that cannot be unpickled then on receiver's side (it does not contain my own classes in PYTHONPATH).

The issue happens only when I use recommened way of passing parameters:

class LibInfo( object ):
  """Library info taken from xml"""
  def __init__(self, libName=None, xmlName=None, packName=None):
    self.libName = libName
    self.xmlName = xmlName
    self.packName = packName

  def __repr__(self):
    return "L=%s X=%s P=%s" % (self.libName,self.xmlName,self.packName)

myObj = LibInfo("1", "2", "3")
logging.info("Simple data: %s", myObj)

Traceback (most recent call last):
  File "/opt/python2.7/logging/handlers.py", line 563, in emit
    s = self.makePickle(record)
  File "/opt/python2.7/logging/handlers.py", line 533, in makePickle
    s = cPickle.dumps(record.__dict__, 1)
PicklingError: Can't pickle <class 'LibInfo'>: attribute lookup LibInfo failed


# these two lines work properly:
logging.info("Simple data: %s", str(myObj) )
logging.info("Simple data: %s" % myObj)


This would be not that critical: I could convert all passed parameters to strings. The issue is with external libraries. That I cannot control.

I think SocketHandler should make record with all parameters resolved to final string.
msg157088 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-03-29 19:19
New changeset eda0ae0d2c68 by Vinay Sajip in branch '2.7':
Closes #14436: Convert msg + args to string before pickling.
http://hg.python.org/cpython/rev/eda0ae0d2c68

New changeset cd8347e15f62 by Vinay Sajip in branch '3.2':
Closes #14436: Convert msg + args to string before pickling.
http://hg.python.org/cpython/rev/cd8347e15f62

New changeset 86a1f92c66b3 by Vinay Sajip in branch 'default':
Closes #14436: merged fix from 3.2.
http://hg.python.org/cpython/rev/86a1f92c66b3
msg237903 - (view) Author: Nathan Jensen (Nathan Jensen) Date: 2015-03-11 21:00
I'm not sure this was fixed in an optimal way.

We have a set of processes using the SocketHandler to send log records to a single log process.  Some of these log records have the msg attribute as a dictionary that contained a variety of extra information about the event that was being logged.  After this change, our process receiving the events and logging to a file stopped working correctly because it was expecting a dictionary for the msg but was always receiving a string.  (I have since made it smarter).

The fix for this ticket makes sense when you don't have a guarantee of being able to unpickle the msg on the receiving end, but it also limits some of the adaptability to pass objects using the SocketHandler.

Some possible improvements:
1. Add a flag to SocketHandler on whether or not it should force the msg to a string
2. If it's it a built-in picklable type, don't force to string, else force msg to string

Suggestion 2 has some drawbacks in that you'd have to loop over lists, dictionaries, etc to verify everything inside there is also picklable.  Also it would prevent you from sending custom objects across.
History
Date User Action Args
2015-03-11 21:00:10Nathan Jensensetnosy: + Nathan Jensen
messages: + msg237903
2012-03-29 19:19:04python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg157088

resolution: fixed
stage: resolved
2012-03-29 11:44:48r.david.murraysetnosy: + vinay.sajip
2012-03-29 10:03:11Zbigniew.Kacprzakcreate