Considering both HAProxy and Benthos can use rfc5424 log format, where fields are explicitly named, we could avoid parsing (with grok) the syslog messages and having a nice improvement in Benthos performances.
Description
Description
Details
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| In Progress | Fabfur | T351117 Move analytics log from Varnish to HAProxy | |||
| Declined | Fabfur | T358109 Install new Benthos instance on cp hosts | |||
| Declined | Fabfur | T360454 Better Benthos performances | |||
| Resolved | Fabfur | T365718 Switch HAProxy/Benthos to rfc5424 | |||
| Declined | Fabfur | T366031 Upgrade Benthos package on cp hosts | |||
| Resolved | Fabfur | T367756 Upgrade hosts to haproxy 2.8.10 | |||
| Resolved | Vgutierrez | T367963 Investigate increase in CD termination state after upgrading eqsin/ulsfo to HAProxy 2.8.10 |
Event Timeline
Comment Actions
Change #1035440 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos:cache: switch to rfc5424 format
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-05-27T13:18:13Z] <fabfur> disabling puppet on A:cp to safely apply https://gerrit.wikimedia.org/r/c/operations/puppet/ /1035440 (T365718)
Comment Actions
Change #1035440 merged by Fabfur:
[operations/puppet@production] benthos:cache: switch to rfc5424 format
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-05-27T13:29:00Z] <fabfur> enabled puppet on cp4037 to apply https://gerrit.wikimedia.org/r/c/operations/puppet/ /1035440 (T365718)
Comment Actions
Change #1036276 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos:cache: fix processing syntax
Comment Actions
Change #1036276 merged by Fabfur:
[operations/puppet@production] benthos:cache: fix processing syntax
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-05-27T15:01:21Z] <fabfur> enable puppet on A:cp (T365718)
Comment Actions
Change #1036711 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos:cache: switch to rfc5424 format
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-05-29T13:16:03Z] <fabfur> temporary disabling puppet on A:cp to rollout https://gerrit.wikimedia.org/r/c/operations/puppet/ /1036711 (T365718)
Comment Actions
Change #1036711 merged by Fabfur:
[operations/puppet@production] benthos:cache: switch to rfc5424 format
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-05-29T14:37:31Z] <fabfur> enabled puppet on A:cp as https://gerrit.wikimedia.org/r/c/operations/puppet/ /1036711 has been reverted (not applied anywhere but cp4037) (T365718)
Comment Actions
This has been reverted due to a bug in HAProxy regarding how log variables are escaped in the rfc5424 format using the E flag.
See:
- https://git.haproxy.org/?p=haproxy-2.6.git;a=commitdiff;h=bb8b763
- https://git.haproxy.org/?p=haproxy-2.6.git;a=commitdiff;h=52d92de
Our experience with this is unterminated log lines (especially for long/complex query strings) that are then "merged" with next log lines and failed to parse by Benthos.
Even if this happens just on a fraction of requests, it's enough to temporary stop the rolling out of this change and wait for the fix to be backported to our version or find a workaround.
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-06-12T13:49:07Z] <fabfur> depooled cp4037 to test benthos/haproxy configuration (T365718)
Comment Actions
Change #1047442 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos:cache: switch to rfc5424 format
Comment Actions
Change #1047442 merged by Fabfur:
[operations/puppet@production] benthos:cache: switch to rfc5424 format
Comment Actions
Change #1047536 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos:cache: fixed typo in field name
Comment Actions
Change #1047536 merged by Fabfur:
[operations/puppet@production] benthos:cache: fixed typo in field name
Comment Actions
Change #1047545 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos:cache: delete message if sequence number is missing
Comment Actions
Change #1047545 merged by Fabfur:
[operations/puppet@production] benthos:cache: delete message if sequence number is missing
Comment Actions
Change #1048024 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] haproxy:cache: raise max uri len to 2048B
Comment Actions
Change #1048024 merged by Fabfur:
[operations/puppet@production] haproxy:cache: raise max uri len to 2048B
Comment Actions
Change #1049498 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos:cache: added catch resource to log errors in parse_log
Comment Actions
Change #1049498 merged by Fabfur:
[operations/puppet@production] benthos:cache: added catch resource to log errors in parse_log
Comment Actions
Change #1049944 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos:cache: moving parse_log directive to input
Comment Actions
Change #1049944 merged by Fabfur:
[operations/puppet@production] benthos:cache: moving parse_log directive to input
Comment Actions
Change #1050258 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos:cache: using tcp socket to send syslog messages from haproxy
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-06-27T10:28:10Z] <fabfur> disable puppet on A:cp-ulsfo to apply selectively https://gerrit.wikimedia.org/r/c/operations/puppet/ /1050258 (T365718)
Comment Actions
Change #1050258 merged by Fabfur:
[operations/puppet@production] benthos:cache: using tcp socket to send syslog messages from haproxy
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-06-27T10:43:52Z] <fabfur> re-enabling puppet on A:cp-text_ulsfo (reverted https://gerrit.wikimedia.org/r/c/operations/puppet/ /1050297) (T365718)
Comment Actions
Change #1050361 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos:cache: using tcp proto for syslog messages from haproxy
Comment Actions
Change #1050361 merged by Fabfur:
[operations/puppet@production] benthos:cache: using tcp proto for syslog messages from haproxy
Comment Actions
Change #1050401 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos:cache: apparently linter has lost the ability to skip envvar
Comment Actions
Change #1050401 merged by Fabfur:
[operations/puppet@production] benthos:cache: apparently linter has lost the ability to skip envvar
Comment Actions
Change #1051198 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos:cache: encode referer field as hex
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-07-04T08:24:49Z] <fabfur> temporary disable puppet on A:cp-ulsfo to apply https://gerrit.wikimedia.org/r/c/operations/puppet/ /1051198 (T365718)
Comment Actions
Change #1051198 merged by Fabfur:
[operations/puppet@production] benthos:cache: encode problematic fields as b64url
Comment Actions
Change #1052052 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos:cache: fix typo in bloblang assignment
Comment Actions
Change #1052052 merged by Fabfur:
[operations/puppet@production] benthos:cache: fix typo in bloblang assignment
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-07-04T08:45:33Z] <fabfur> enable puppet on A:cp-ulsfo (T365718)