Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Protobuf consumer support #353

Open
sbillet opened this issue Nov 5, 2021 · 2 comments
Open

Protobuf consumer support #353

sbillet opened this issue Nov 5, 2021 · 2 comments

Comments

@sbillet
Copy link

sbillet commented Nov 5, 2021

First of all, thank you for providing this immensely useful tool!

Here comes a feature request.

Since April 2020 the Confluent Kafka platform has "first-class citizen" support of Protobuf protocol and schemas in all components (source). Producing/consuming data and the interaction with the Schema Registry works in the same transparent way as for Avro format.

It would be fantastic if kafkacat could provide the same native support for consuming Protobuf messages as for Avro messages.

With that feature, it should be possible to write

kafkacat -C -b localhost:9092 \
-t proto-topic \
-s key=s -s value=proto \
-r http://localhost:8081

and get the topic messages printed nicely because Kafkacat got the schema from the registry and did the deserializing.

This would be much nicer for manual interaction than having to pipe the output to protoc as described in #72

@VanceLongwill
Copy link

I've written a small CLI in Rust to do just this with a similar interface to kcat. It's called milena and it supports consuming/decoding and producing/encoding protobuf kafka messages to and from JSON for ease of use

@oberbichler
Copy link

protosaurus can automatically deserialize the output of kcat using the corresponding protos from a schema registry:

kcat -C -e -F <kafka.config> -t <topic> -f "%o\\n%k\\n%R%s" | protosaurus - --schema-registry <url>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants