Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pluggable Cluster Backends - Part 3 #4037

Open
wants to merge 7 commits into
base: topic/awelzel/pluggable-cluster-backends-part2
Choose a base branch
from

Conversation

awelzel
Copy link
Contributor

@awelzel awelzel commented Nov 14, 2024

Mostly logging related changes:

  • WriterFrontends have a new LogWriteHeader
  • Non-broker cluster backends re-use the WriterFrontend log write buffering and flushing
    • That one is a bit hairy (logging/WriterFrontend: Add logic for non-broker cluster backends)
  • Non-broker cluster backends will use a new WritesFromRemote() entry point in the logging manager
  • Introduces a log serializer using SerialTypes named binary-serialization-format. Similar to broker, but the whole batch uses SerialTypes logic. This can be replaced in the future with something more efficient.

@awelzel awelzel force-pushed the topic/awelzel/pluggable-cluster-backends-part3 branch from 484c706 to 3f4e18a Compare November 14, 2024 18:12
@awelzel awelzel marked this pull request as ready for review November 14, 2024 18:34
@awelzel awelzel force-pushed the topic/awelzel/pluggable-cluster-backends-part2 branch from 551b01f to d183a46 Compare November 15, 2024 13:33
@awelzel awelzel force-pushed the topic/awelzel/pluggable-cluster-backends-part3 branch from 3f4e18a to 135af2a Compare November 15, 2024 13:33
...with this change, it'll be possible to identify WriterFrontend's
based on (stream, filter_name, path) pairs in addition to (stream,
writer, path) pairs.
The header captures the enum values as well as the fields
If cluster::backend isn't broker_mgr, use the WriterFrontend's buffering
logic and send a whole batch of log writes during FlushWriteBuffer().
This is a different path than broker's own logging logic.

Preferably we adapt broker to a model where it isn't
buffering either.
This is a serializer for log records that is using SerialTypes
for serializing and un-serializing rather. Essentially, this is
similar to what broker does except for the envelope.
@awelzel awelzel force-pushed the topic/awelzel/pluggable-cluster-backends-part2 branch from d183a46 to 04332bf Compare November 15, 2024 13:43
@awelzel awelzel force-pushed the topic/awelzel/pluggable-cluster-backends-part3 branch from 135af2a to 947ee29 Compare November 15, 2024 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant