Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kafkacat shows (all?) old events after connection issues #300

Open
basvs opened this issue Mar 3, 2021 · 0 comments
Open

kafkacat shows (all?) old events after connection issues #300

basvs opened this issue Mar 3, 2021 · 0 comments

Comments

@basvs
Copy link

basvs commented Mar 3, 2021

Triggered in both Consume mode (-C) and High-level KafkaConsumer mode (-G)

To reproduce, I started kafkacat with these options:
-b kafka-0.local:19092,kafka-1.local:19092,kafka-2.local:19092
-u
-X api.version.request=true
-E
-v
-o end
-J
-G my-group
my-events

I sent the output to a script which checks the timestamp in the payload message; if the message is more than a few minutes old, the script will print the full json message and abort.

After a few hours this resulted in:

…
% Reached end of topic my-events [5] at offset 2346458
% Reached end of topic my-events [1] at offset 2345329
% Reached end of topic my-events [7] at offset 2346438
% Reached end of topic my-events [8] at offset 2344935
% Reached end of topic my-events [0] at offset 2348516
% Reached end of topic my-events [9] at offset 2346675
% Reached end of topic my-events [7] at offset 2346439
% Reached end of topic my-events [8] at offset 2344936
% Reached end of topic my-events [2] at offset 2344322
% Reached end of topic my-events [4] at offset 2344355
% ERROR: Local: Broker transport failure: 192.168.0.11:19092/1: Disconnected (after 10583607ms in state UP)
%4|1614261329.279|SESSTMOUT|rdkafka#consumer-1| [thrd:main]: Consumer group session timed out (in join-state started) after 10446 ms without a successful response from the group coordinator (broker 1, last error was Success): revoking assignment and rejoining group
% Group my-group rebalanced (memberid ): revoked: my-events [0], my-events [1], my-events [2], my-events [3], my-events [4], my-events [5], my-events [6], my-events [7], my-events [8], my-events [9]
%5|1614261332.426|REQTMOUT|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Timed out HeartbeatRequest in flight (after 10602ms, timeout #0)
%4|1614261332.426|REQTMOUT|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
% ERROR: Local: Timed out: GroupCoordinator: 1 request(s) timed out: disconnect (after 10597187ms in state UP)
% ERROR: Local: Broker transport failure: 192.168.0.11:19092/1: Disconnected (after 15162ms in state UP)
% Group my-group rebalanced (memberid rdkafka-35e421cb-8b46-49f1-a019-8596c408394c): assigned: my-events [0], my-events [1], my-events [2], my-events [3], my-events [4], my-events [5], my-events [6], my-events [7], my-events [8], my-events [9]
% Reached end of topic my-events [0] at offset 1959560
% Reached end of topic my-events [1] at offset 1956571
% Reached end of topic my-events [2] at offset 1957016
% Reached end of topic my-events [3] at offset 1955765
% Reached end of topic my-events [4] at offset 1956833
% Reached end of topic my-events [5] at offset 1958721
% Reached end of topic my-events [6] at offset 1986343
% Reached end of topic my-events [7] at offset 1959190
% Reached end of topic my-events [8] at offset 1958125
% Reached end of topic my-events [9] at offset 1986351
% ERROR: Local: Broker transport failure: 192.168.0.11:19092/1: Connect to ipv4#192.168.0.11:19092 failed: No route to host (after 10276ms in state CONNECT)
% Reached end of topic my-events [3] at offset 1955766
% ERROR: Local: Broker transport failure: 192.168.0.10:19092/0: Disconnected (after 42988ms in state UP)
% ERROR: Local: Broker transport failure: 192.168.0.10:19092/0: Disconnected (after 117ms in state UP)
% ERROR: Local: Broker transport failure: GroupCoordinator: Disconnected (after 28165ms in state UP)
% ERROR: Local: Broker transport failure: 192.168.0.10:19092/0: Connect to ipv4#192.168.0.10:19092 failed: Connection refused (after 7ms in state CONNECT)
% ERROR: Local: Broker transport failure: GroupCoordinator: Connect to ipv4#192.168.0.10:19092 failed: Connection refused (after 7ms in state CONNECT)
received old event!
{"topic":"my-events","partition":6,"offset":1986343,"tstype":"create","ts":1613640430344,"broker":-1,"key":null,"payload":"my event with timestamp 1613640430.344110"}

This timestamp was from 5 days ago. Way before the -o end I specified when kafkacat started

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant