Description
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| In Progress | Fabfur | T351117 Move analytics log from Varnish to HAProxy | |||
| Declined | Fabfur | T358109 Install new Benthos instance on cp hosts | |||
| Declined | Fabfur | T360454 Better Benthos performances | |||
| Resolved | Fabfur | T365718 Switch HAProxy/Benthos to rfc5424 | |||
| Resolved | Fabfur | T367756 Upgrade hosts to haproxy 2.8.10 | |||
| Resolved | Vgutierrez | T367963 Investigate increase in CD termination state after upgrading eqsin/ulsfo to HAProxy 2.8.10 |
Event Timeline
Change #1047483 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] hiera: test downgrading haproxy on cp5017
Change #1047483 merged by Fabfur:
[operations/puppet@production] hiera: test downgrading haproxy on cp5017
Change #1047492 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] hiera: test upgrading cp5017 to haproxy 2.8.9
Change #1047492 merged by Fabfur:
[operations/puppet@production] hiera: test upgrading cp5017 to haproxy 2.8.9
this is caused a bug in the mtail regex used to parse haproxy logs, on haproxy 2.6.17 http status gets reported as -1:
2024-06-24T13:46:52.422444 00:00 cp7001 haproxy[2780368]: 180684 -1 0 0 -1 {es.wikipedia.org} {} CDbut the regex expects a 3 figures number from 100 to 599:
(?P<http_status_family>[1-5])\d\d
Change #1049183 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):
[operations/puppet@production] mtail: Allow http status -1 on cache_haproxy
Change #1049183 merged by Vgutierrez:
[operations/puppet@production] mtail: Allow http status -1 on cache_haproxy
cache_haproxy.mtail failed to accept -1 as an HTTP status code, under reporting CD and CR termination states.
Mentioned in SAL (#wikimedia-operations) [2024-06-24T15:43:51Z] <vgutierrez> updated termination_state cache haproxy metrics, expect higher CD and CR rates - T367963