Ticket description has been updated to reflect the outcome of latest meeting with @gmodena and @Ahoelzl .

The CPUAffinity property has been set manually on cp4037 and cp3066 using /etc/systemd/system/haproxykafka.service.d/override.conf to understand if that's a possible improvement factor. If this shows benefits it can be brought by puppet relatively easily

Package has been added to apt repository:

Hi @RobH , do you need anything from us (Traffic) for this? Can we help?

We've no issues in changing topic names even at the latest moment, as they will be completely configurable and they just need a haproxykafka service reload, so let me know whatever option is best for you!

In T374473#10136693, @gmodena wrote:

Hi,

Are you planning to start shipping to Kafka anytime soon? It would be great to touch base and iron out the timelines in case, just to stay on the same page.

Hosts cp203[5-6] downtimed and depooled

Superseded by T370668

Superseded by T370668

Superseded by T370668

Superseded by T370668

Superseded by T370668

Superseded by T370668

Superseded by T370668

Superseded by T370668

Superseded by T370668

In T372344#10059545, @Vgutierrez wrote:

This would need some filtering and we should be very careful on how it's implemented to not trigger a big performance degradation. I'd say that stdout/file is ruled out as we don't want PII hitting the disk (I include stdout here cause systemd persists it to disk)

ACK, now I remember the talk about that

This has been completed and verified on hosts in ulsfo

In T370741#10041302, @Vgutierrez wrote:

we had some alerts ongoing during the weekend due to this task:

FIRING: SystemdUnitFailed: wmf_auto_restart_benthos@haproxy_cache.service on cp4037:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname=SystemdUnitFailed

after checking that on the PCC run for https://gerrit.wikimedia.org/r/c/operations/puppet/ /1057823 /lib/systemd/system/wmf_auto_restart_benthos@haproxy_cache.service wasn't on the change catalog, I removed it manually:

sudo -i rm -f /lib/systemd/system/wmf_auto_restart_benthos@haproxy_cache.service
sudo -i systemctl daemon-reload 
sudo -i systemctl reset-failed

double checked that puppet won't recreate the systemd service unit and alerts are now gone

Do you need access to private data? Because the procedure will be different in case... have a look at https://wikitech.wikimedia.org/wiki/Data_Platform/Data_access#What_access_should_I_request? pls

Do you need access to private data? Because the procedure will be different in case... have a look at https://wikitech.wikimedia.org/wiki/Data_Platform/Data_access#What_access_should_I_request? pls

As per procedure, requesting confirmation also from the Data-Engineering group

As per procedure, requesting confirmation also from the Data-Engineering group

Hi @NBaca-WMF @thcipriani, could you confirm this request please?

Puppet run on all cp-ulsfo hosts has been completed, HAProxy configuration now doesn't send logs anymore to Benthos.
In the next days we'll remove (manually) packages and configuration files from the hosts.