Message267487
Snippet code is below:
from kafka import KafkaConsumer
from kafka.client import KafkaClient
from kafka.consumer import SimpleConsumer
consumer = KafkaConsumer('eventdetails_ingestion' , group_id='1',bootstrap_servers=‘xxxx:9092', max_partition_fetch_bytes=10555555)
for msg in consumer:
try:
jValue = json.loads(str(msg.value))
except ValueError:
fileErr.write(str(msg.value)+"\n")
Steps:
We send/produce large sets of messages to Kafka of around 20 to 30 KB size each messages in JSON format and producing around 200 messages / sec for 1 hour duration. We have 3 Kafka Brokers running and I am trying to consume the messages from these 3 Kafka Brokers from the same topic using the above code. The problem is that sometimes some of the messages gets truncated, I am not sure why it happen ? |
|
Date |
User |
Action |
Args |
2016-06-06 00:06:03 | rgowda@threatmetrix.com | set | recipients:
+ rgowda@threatmetrix.com |
2016-06-06 00:06:03 | rgowda@threatmetrix.com | set | messageid: <1465171563.42.0.101406692034.issue27237@psf.upfronthosting.co.za> |
2016-06-06 00:06:03 | rgowda@threatmetrix.com | link | issue27237 messages |
2016-06-06 00:06:02 | rgowda@threatmetrix.com | create | |
|