New software: haproxykafka
Open, Needs TriagePublic
Actions

Assigned To

Authored By

	Fabfur
	Jul 22 2024, 4:10 PM

Description

Umbrella task to follow development of haproxykafka, a software meant to replace Benthos (currently just in testing phase in ulsfo) to parse HAProxy logs over socket and send to specific Kafka topic

Details

Subject	Repo	Branch	Lines /-
Renamed log field for pipeline migration (haproxykafka)	operations/puppet	production	1 -1
cache:haproxy: hardcode $schema field	operations/puppet	production	1 -1
hiera: continue haproxykafka tests on cp4037	operations/puppet	production	2 -2
hiera: disabling haproxy logging to socket (haproxykafka)	operations/puppet	production	2 -2
cache:haproxy: fixed the haproxykafka uds path	operations/puppet	production	1 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
In Progress	Fabfur	T351117 Move analytics log from Varnish to HAProxy
Open	Fabfur	T370668 New software: haproxykafka
Resolved	Fabfur	T372338 haproxykafka: feature: Add prometheus metrics
Declined	Fabfur	T372341 haproxykafka: feature: Read from TCP UDS socket
Resolved	Fabfur	T372339 haproxykafka: feature: add ability to add/remove/modify fields
Resolved	Fabfur	T372342 haproxykafka: feature: Configuration file
Open	Fabfur	T372344 haproxykafka: feature: Ability to print structured messages to stdout
Open	Fabfur	T374128 haproxykafka features
Open	Fabfur	T374473 Prepare puppet configuration to send haproxy logs to haproxykafka socket
Open	Fabfur	T374696 Enable prometheus metrics scraping for haproxykafka

Event Timeline

Fabfur created this task.Jul 22 2024, 4:10 PM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptJul 22 2024, 4:10 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

I might be out of my league here, but have yall considered the haproxy Stream Processing Offload Engine?

It looks like a built in way to hook into haproxy requests and do whatever you need. This might be more efficient than listening to a UDP socket with syslog formatted messages. There looks to be SPOE agent libraries in C. Maybe you could even re-use some varnishkafka code to bridge between the SPOE protocol and librdkafka?

Fabfur updated the task description. (Show Details)Jul 22 2024, 4:36 PM

https://github.com/negasus/haproxy-spoe-go could be handy if we go down that road and we don't want to get dirty writing C code :)

In T370668#10003489, @Ottomata wrote:

I might be out of my league here, but have yall considered the haproxy Stream Processing Offload Engine?

It looks like a built in way to hook into haproxy requests and do whatever you need. This might be more efficient than listening to a UDP socket with syslog formatted messages. There looks to be SPOE agent libraries in C. Maybe you could even re-use some varnishkafka code to bridge between the SPOE protocol and librdkafka?

Hi Andrew, this is definitely something worth considering, but I would need a serious help w/ C.
ATM we decided to go down the road of rewriting atskafka in it's semplicity to adapt it to HAProxy with structured log format (RFC5424),

In the meantime I'll investigate into SPOE... thanks!

I would need a serious help w/ C.

Ya, me too! Perhaps the SPOE go lib @Vgutierrez mentioned might be easier?

ATM we decided to go down the road of rewriting atskafka in it's semplicity to adapt it to HAProxy with structured log format (RFC5424),

If it works, then it works! I remember when we hired Magnus Edenhill to write varnishkafka, he did a lot of very specific performance and memory focused tuning to make sure stuff would work well. (and um, and I wonder if we could contract him out again? :D Probably not; he's more famous now.)

In either case, we'll be using librdkafka underneath. Perhaps RFC5424 format vs SPOP won't really make much of a performance difference.

VirginiaPoundstone moved this task from Incoming to Radar (other teams) on the Data Products board.Jul 26 2024, 4:14 PM

untagging o11y since this is unrelated

CDanis subscribed.Aug 12 2024, 7:28 PM

Fabfur added a subtask: T372339: haproxykafka: feature: add ability to add/remove/modify fields.Aug 12 2024, 7:32 PM

Fabfur closed subtask T372341: haproxykafka: feature: Read from TCP UDS socket as Declined.Aug 12 2024, 7:38 PM

Fabfur mentioned this in T358109: Install new Benthos instance on cp hosts.Aug 21 2024, 9:56 AM

Fabfur mentioned this in T365968: Install benthos on single esams host to check performances under higher load.

Fabfur mentioned this in T364379: Benthos loses messages when under high load.

Fabfur mentioned this in T360454: Better Benthos performances.

Fabfur mentioned this in T361845: Add metrics to Benthos.

Fabfur mentioned this in T359178: Check statsv and eventlogging VarnishKafka instances.Aug 21 2024, 9:58 AM