Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose Kafka metadata w/ Kafka-sourced messages #125

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

charlie
Copy link
Contributor

@charlie charlie commented Jun 11, 2020

Apologies if this idea has been discussed and rejected before. I feel like maybe it has. Or if there's already some other way to do this, and I just couldn't see it.

Intended usage:

offset := int64(-1)
if withMetadata, ok := msg.(kafka.MessageWithMetadata); ok {
	offset = withMetadata.GetMetadata().Offset
}

@mjgarton
Copy link
Contributor

I think this currently sits outside the goals, or perhaps even goes against the goals of substrate, but I'm interested in hearing more about why you want to do this in your particular use case, so that I can try think about alternate solutions. @charlie

@charlie
Copy link
Contributor Author

charlie commented Jun 11, 2020

To describe it narrowly, the motivation is to provide a unique identifier for each message within a topic.

In practice, the goal is to project the entire contents of a topic into an Elasticsearch index, using these identifiers as the document IDs. Note that while the contents of the messages contain IDs, these are not guaranteed to be unique, as the same message contents (i.e. the same ID) may be published to the topic more than once (by misbehaving publishers). I want the projection to reflect the existence of both messages in the topic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants