This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author rgowda@threatmetrix.com
Recipients rgowda@threatmetrix.com
Date 2016-06-06.00:06:02
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1465171563.42.0.101406692034.issue27237@psf.upfronthosting.co.za>
In-reply-to
Content
Snippet code is below:
from kafka import KafkaConsumer
from kafka.client import KafkaClient
from kafka.consumer import SimpleConsumer

consumer = KafkaConsumer('eventdetails_ingestion' , group_id='1',bootstrap_servers=‘xxxx:9092', max_partition_fetch_bytes=10555555)
for msg in consumer:
    try:
        jValue = json.loads(str(msg.value))
   except ValueError:
        fileErr.write(str(msg.value)+"\n")

Steps:
We send/produce large sets of messages to Kafka of around 20 to 30 KB size each messages in JSON format and producing around 200 messages / sec for 1 hour duration. We have 3 Kafka Brokers running and I am trying to consume the messages from these 3 Kafka Brokers from the same topic using the above code. The problem is that sometimes some of the messages gets truncated, I am not sure why it happen ?
History
Date User Action Args
2016-06-06 00:06:03rgowda@threatmetrix.comsetrecipients: + rgowda@threatmetrix.com
2016-06-06 00:06:03rgowda@threatmetrix.comsetmessageid: <1465171563.42.0.101406692034.issue27237@psf.upfronthosting.co.za>
2016-06-06 00:06:03rgowda@threatmetrix.comlinkissue27237 messages
2016-06-06 00:06:02rgowda@threatmetrix.comcreate