You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi, I'm consuming data from a secure kafka deployed in client's env.
I can consume data without -G which means it runs as low-level consumer,
however in order to figure out some issues (another story), I need to test it with high-level consumer and it ends up with error.
join state changed from wait-sync -> wait-assign-rebalance_cb (v1, state up)
then it just stop. got no clue with that.
Data can be consumed with Spark which uses high-level consumer as well.
Thus, I am not sure about the root cause is in server side or not.
What should I do to make it work ?
the version of kafka and librdkaka:
$ kafkacat -V
kafkacat - Apache Kafka producer and consumer tool
https://github.com/edenhill/kafkacat
Copyright (c) 2014-2020, Magnus Edenhill
Version 1.7.0 (Transactions, IncrementalAssign, librdkafka 0.11.5 builtin.features=gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins)
part of log shows:
$ kafkacat -b hdp2-dkk0001.company.com:9092,hdp2-dkk0002.company.com:9092,hdp2-dkk0003.company.com:9092,hdp2-dkk0004.office.corp.[580/2733]m:9092,hdp2-dkk0005.company.com:9092,hdp2-dkk0006.company.com:9092,hdp2-dkk0007.company.com:9092,hdp2-dkk0008.company.com:9092,hdp2-dkk0009.company.com:9092,hdp2-dkk0010.company.com:9092 -G spark-executor-consumergroup -v -d cgrp,broker,protocol,fetch data_topic -X security.protocol=SASL_PLAINTEXT -X sasl.kerberos.keytab=/etc/hdp-dmp_ght.keytab -X [email protected] -X sasl.kerberos.service.name=kafka
%7|1658455001.772|MEMBERID|rdkafka#consumer-1| [thrd:app]: Group "spark-executor-consumergroup": updating member id "(not-set)" -> ""
%7|1658455001.773|BRKMAIN|rdkafka#consumer-1| [thrd::0/internal]: :0/internal: Enter main broker thread
%7|1658455001.773|BRKREASSIGN|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup" management reassigned from broker (none) to :0/internal
%7|1658455001.773|STATE|rdkafka#consumer-1| [thrd::0/internal]: :0/internal: Broker changed state INIT -> UP
%7|1658455001.773|CGRPSTATE|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup" changed state init -> wait-broker (v1, join-state init)
%7|1658455001.773|BROKER|rdkafka#consumer-1| [thrd:app]: sasl_plaintext://hdp2-dkk0001.company.com:9092/bootstrap: Added new broker with NodeId -1
%7|1658455001.773|BRKASSIGN|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup" management assigned to broker :0/internal
%7|1658455001.773|BRKMAIN|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0001.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0001.company.com:9092/bootstrap: Enter main brokerd thread
%7|1658455001.773|CONNECT|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0001.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0001.company.com:9092/bootstrap: broker in state INIT connecting
....
%7|1658455001.774|CONNECT|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0009.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0009.company.com:9092/bootstrap: broker in state INIT connecting
%7|1658455001.774|CGRPOP|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup" received op SUBSCRIBE (v0) in state wait-broker (join state init, v1 vs 0)
%7|1658455001.774|SUBSCRIBE|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup": subscribe to new subscription of 1 topics (join state init)
%7|1658455001.774|UNSUBSCRIBE|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup": unsubscribe from current unset subscription of 0 topics (leave group=no, join state init, v1)
%7|1658455001.774|GRPLEADER|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup": resetting group leader info: unsubscribe
%7|1658455001.774|CGRPJOINSTATE|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup" changed join state init -> wait-unassign (v1, state wait-broker)
%7|1658455001.774|UNASSIGN|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup": unassign done in state wait-broker (join state wait-unassign): without new assignment: unassign (no previous assignment)
%7|1658455001.774|CGRPJOINSTATE|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup" changed join state wait-unassign -> init (v1, state wait-broker) % Waiting for group rebalance
%7|1658455001.774|CGRPQUERY|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup": no broker available for coordinator query: intervaled in state wait-broker
%7|1658455001.775|CONNECT|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0005.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0005.company.com:9092/bootstrap: Connecting to ipv4#10.34.163.96:9092 (sasl_plaintext) with socket 27
%7|1658455001.775|STATE|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0005.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0005.company.com:9092/bootstrap: Broker changed state INIT -> CONNECT
...
%7|1658455001.831|SEND|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0006.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0006.company.com:9092/bootstrap: Sent MetadataRequest (v2, 25 bytes @ 0, CorrId 3)
%7|1658455001.871|RECV|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0006.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0006.company.com:9092/bootstrap: Received MetadataResponse (v2, 516 bytes, CorrId 3, rtt 39.78ms)
%7|1658455001.871|STATE|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0005.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0005.company.com:9092/bootstrap: Broker changed state AUTH -> UP
%7|1658455001.871|CLUSTERID|rdkafka#consumer-1| [thrd:main]: sasl_plaintext://hdp2-dkk0006.company.com:9092/bootstrap: ClusterId update "" -> "aQRChANaRTORz19uCdIEgw"
%7|1658455001.871|CONTROLLERID|rdkafka#consumer-1| [thrd:main]: sasl_plaintext://hdp2-dkk0006.company.com:9092/bootstrap: ControllerId update -1 -> 2304
%7|1658455001.871|SUBSCRIPTION|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup": no topics in metadata matched subscription
%7|1658455001.871|UPDATE|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0006.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0006.company.com:9092/bootstrap: NodeId changed from -1 to 2304
%7|1658455001.871|UPDATE|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0006.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0006.company.com:9092/2304: Name changed from sasl_plaintext://hdp2-dkk0006.company.com:9092/bootstrap to sasl_plaintext://hdp2-dkk0006.company.com:9092/2304
%7|1658455001.871|LEADER|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0006.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0006.company.com:9092/2304: Mapped 0 partition(s) to broker
%7|1658455001.871|STATE|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0006.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0006.company.com:9092/2304: Broker changed state UP -> UPDATE
%7|1658455001.871|UPDATE|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0005.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0005.company.com:9092/bootstrap: NodeId changed from -1 to 730
%7|1658455001.871|STATE|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0006.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0006.company.com:9092/2304: Broker changed state UPDATE -> UP
%7|1658455001.871|UPDATE|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0005.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0005.company.com:9092/730: Name changed from sasl_plaintext://hdp2-dkk0005.company.com:9092/bootstrap to sasl_plaintext://hdp2-dkk0005.company.com:9092/730
%7|1658455001.871|LEADER|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0005.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0005.company.com:9092/730: Mapped 0 partition(s) to broker
%7|1658455001.871|BRKREASSIGN|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup" management reassigned from broker :0/internal to sasl_plaintext://hdp2-dkk0006.company.com:9092/2304
%7|1658455001.871|BRKUNASSIGN|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup" management unassigned from broker handle :0/internal
%7|1658455001.871|STATE|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0005.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0005.company.com:9092/730: Broker changed state UP -> UPDATE
%7|1658455001.871|BRKASSIGN|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup" management assigned to broker sasl_plaintext://hdp2-dkk0006.company.com:9092/2304
%7|1658455001.871|STATE|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0005.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0005.company.com:9092/730: Broker changed state UPDATE -> UP
%7|1658455001.871|CGRPQUERY|rdkafka#consumer-1| [thrd:main]: sasl_plaintext://hdp2-dkk0005.company.com:9092/730: Group "spark-executor-consumergroup": querying for coordinator: intervaled in state wait-broker
%7|1658455001.871|SEND|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0005.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0005.company.com:9092/730: Sent GroupCoordinatorRequest (v0, 54 bytes @ 0, CorrId 3)
%7|1658455001.913|RECV|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0005.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0005.company.com:9092/730: Received GroupCoordinatorResponse (v0, 48 bytes, CorrId 3, rtt 41.49ms)
%7|1658455001.913|CGRPCOORD|rdkafka#consumer-1| [thrd:main]: sasl_plaintext://hdp2-dkk0005.company.com:9092/730: Group "spark-executor-consumergroup" coordinator is hdp2-dkk0002.company.com:9092 id 729
%7|1658455001.913|CGRPCOORD|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup" changing coordinator -1 -> 729
%7|1658455001.929|UPDATE|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0004.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0004.company.com:9092/bootstrap: NodeId changed from -1 to 731
%7|1658455001.929|UPDATE|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0004.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0004.company.com:9092/731: Name changed from sasl_plaintext://hdp2-dkk0004.company.com:9092/bootstrap to sasl_plaintext://hdp2-dkk0004.company.com:9092/731
%7|1658455001.929|LEADER|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0004.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0004.company.com:9092/731: Mapped 0 partition(s) to broker
%7|1658455001.933|STATE|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0004.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0004.company.com:9092/731: Broker changed state AUTH -> UP
...
%7|1658455002.239|SEND|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0003.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0003.company.com:9092/727: Sent MetadataRequest (v2, 25 bytes @ 0, CorrId 3)
%7|1658455002.281|RECV|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0003.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0003.company.com:9092/727: Received MetadataResponse (v2, 516 bytes, CorrId 3, rtt 41.36ms)
%7|1658455006.773|COMMIT|rdkafka#consumer-1| [thrd:main]: OffsetCommit internal error: Local: No offset stored
%7|1658455006.773|COMMIT|rdkafka#consumer-1| [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored
%7|1658455006.773|UNASSIGN|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET)
%7|1658455011.757|RECV|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0002.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0002.company.com:9092/729: Received JoinGroupResponse (v0, 196 bytes, CorrId 4, rtt 9703.42ms)
%7|1658455011.757|JOINGROUP|rdkafka#consumer-1| [thrd:main]: JoinGroup response: GenerationId 7, Protocol range, LeaderId rdkafka-ed00da91-efdc-4925-9d39-d4cda4adbe4f (me), my MemberId rdkafka-ed00da91-efdc-4925-9d39-d4cda4adbe4f, 1 members in group: (no error)
%7|1658455011.757|MEMBERID|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup": updating member id "" -> "rdkafka-ed00da91-efdc-4925-9d39-d4cda4adbe4f"
%7|1658455011.757|JOINGROUP|rdkafka#consumer-1| [thrd:main]: Elected leader for group "spark-executor-consumergroup" with 1 member(s)
%7|1658455011.757|GRPLEADER|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup": resetting group leader info: JoinGroup response clean-up
%7|1658455011.757|CGRPJOINSTATE|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup" changed join state wait-join -> wait-metadata (v1, state up)
%7|1658455011.757|SEND|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0002.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0002.company.com:9092/729: Sent MetadataRequest (v2, 52 bytes @ 0, CorrId 5)
%7|1658455011.759|RECV|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0002.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0002.company.com:9092/729: Received MetadataResponse (v2, 1978 bytes, CorrId 5, rtt 2.21ms)
%7|1658455011.759|ASSIGN|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup" running range assignment for 1 member(s):
%7|1658455011.759|ASSIGN|rdkafka#consumer-1| [thrd:main]: Member "rdkafka-ed00da91-efdc-4925-9d39-d4cda4adbe4f" (me) with 1 subscription(s):
%7|1658455011.759|ASSIGN|rdkafka#consumer-1| [thrd:main]: data_topic [-1]
%7|1658455011.759|ASSIGN|rdkafka#consumer-1| [thrd:main]: range: Topic data_topic with 34 partition(s) and 1 subscribing member(s)
%7|1658455011.759|ASSIGN|rdkafka#consumer-1| [thrd:main]: range: Member "rdkafka-ed00da91-efdc-4925-9d39-d4cda4adbe4f": assigned topic data_topic partitions 0..33
%7|1658455011.759|ASSIGN|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup" range assignment for 1 member(s) finished in 0.031ms:
%7|1658455011.759|ASSIGN|rdkafka#consumer-1| [thrd:main]: Member "rdkafka-ed00da91-efdc-4925-9d39-d4cda4adbe4f" (me) assigned 34 partition(s):
%7|1658455011.759|ASSIGN|rdkafka#consumer-1| [thrd:main]: data_topic [0]
%7|1658455011.759|ASSIGN|rdkafka#consumer-1| [thrd:main]: data_topic [1]
%7|1658455011.759|ASSIGN|rdkafka#consumer-1| [thrd:main]: data_topic [2]
......
%7|1658455011.759|ASSIGN|rdkafka#consumer-1| [thrd:main]: data_topic [32]
%7|1658455011.759|ASSIGN|rdkafka#consumer-1| [thrd:main]: data_topic [33]
%7|1658455011.759|ASSIGNOR|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup": "range" assignor run for 1 member(s)
%7|1658455011.759|CGRPJOINSTATE|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup" changed join state wait-metadata -> wait-sync (v1, state up)
%7|1658455011.759|SEND|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0002.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0002.company.com:9092/729: Sent SyncGroupRequest (v0, 335 bytes @ 0, CorrId 6)
%7|1658455011.764|RECV|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0002.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0002.company.com:9092/729: Received SyncGroupResponse (v0, 183 bytes, CorrId 6, rtt 4.24ms)
%7|1658455011.764|SYNCGROUP|rdkafka#consumer-1| [thrd:main]: SyncGroup response: Success (177 bytes of MemberState data)
%7|1658455011.764|ASSIGN|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup": delegating assign of 34 partition(s) to application rebalance callback on queue rd_kafka_cgrp_new: new assignment
%7|1658455011.764|CGRPJOINSTATE|rdkafka#consumer-1| [thrd:main]: Group "spark-executor-consumergroup" changed join state wait-sync -> wait-assign-rebalance_cb (v1, state up)
%7|1658455011.764|HEARTBEAT|rdkafka#consumer-1| [thrd:main]: sasl_plaintext://hdp2-dkk0002.company.com:9092/729: Heartbeat for group "spark-executor-consumergroup" generation id 7
kafkacat: symbol lookup error: kafkacat: undefined symbol: rd_kafka_rebalance_protocol
%7|1658455011.764|SEND|rdkafka#consumer-1| [thrd:sasl_plaintext://hdp2-dkk0002.company.com:9092/boot]: sasl_plaintext://hdp2-dkk0002.company.com:9092/729: Sent HeartbeatRequest (v0, 104 bytes @ 0, CorrId 7)
Hi, I'm consuming data from a secure kafka deployed in client's env.
I can consume data without
-Gwhich means it runs as low-level consumer,however in order to figure out some issues (another story), I need to test it with high-level consumer and it ends up with error.
join state changed from
wait-sync->wait-assign-rebalance_cb(v1, state up)then it just stop. got no clue with that.
Data can be consumed with Spark which uses high-level consumer as well.
Thus, I am not sure about the root cause is in server side or not.
What should I do to make it work ?
the version of kafka and librdkaka:
part of log shows:
full log: (
-d cgrp,broker,protocol,fetch)kcat_consume.log
The text was updated successfully, but these errors were encountered: