You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have a use case where I wish to count the number of records in a Kafka topic that have not yet been processed by a consumer group. To do this, I get the offsets per partition for the consumer group and then read the messages using kafkacat/kcat.
$ cat num_records.txt \
| awk '{ sum = $1 } END { print "Sum of all records still in topic across all partitions: " sum }'
Sum of all records still in topic across all partitions: 117398
If I do the exact same string of commands using docker run edenhill/kcat:1.7.1 instead of kafkacat above, I get a different sum of records:
$ cat num_records.txt
1412
11869
11566
1692
2377
8403
10734
4040
6568
369
13576
14019
516
6199
3685
9365
3851
6332
$ cat num_records.txt \
| awk '{ sum = $1 } END { print "Sum of all records still in topic across all partitions: " sum }'
Sum of all records still in topic across all partitions: 116573
The latter value is the same one that I got when I tried to process my topic from the given offsets in Python using confluent-kafka, so I'm leaning towards it being the correct one.
Was this an undocumented fix? Or did a setting change between 1.5.0 and 1.7.1 that could influence this behaviour? I checked the release notes but unfortunately could not find much that seems relevant...
The text was updated successfully, but these errors were encountered:
filpano
changed the title
Different (undocumented) behaviour when reading in topic between versions
Different (undocumented?) behaviour when reading topic from offset between versions
Sep 28, 2022
I have a use case where I wish to count the number of records in a Kafka topic that have not yet been processed by a consumer group. To do this, I get the offsets per partition for the consumer group and then read the messages using
kafkacat/kcat
.For example:
I use the following command to process all messages from these given offsets until the log end offset:
Writes something like:
which sums up to:
If I do the exact same string of commands using
docker run edenhill/kcat:1.7.1
instead ofkafkacat
above, I get a different sum of records:The latter value is the same one that I got when I tried to process my topic from the given offsets in Python using
confluent-kafka
, so I'm leaning towards it being the correct one.Was this an undocumented fix? Or did a setting change between
1.5.0
and1.7.1
that could influence this behaviour? I checked the release notes but unfortunately could not find much that seems relevant...The text was updated successfully, but these errors were encountered: