Trying to correlate by when we saw OOM killing of apache by querying the logs with something like

OOM
jsonPayload.reason="OOMKilling"
apache

Shows the following graph which is a reasonably good correlation with the time we saw issues but doesn't explain times on the 25th / 27th unless these were resolved too quickly to trigger a failing uptime check.

We've increased the replicas to 3 and also bumped the memory limit.

The system seems stable as of now (Thursday morning) so I am closing this ticket.

What happened on the way here:

• after having 3 replicas with more memory running we have seen the SQL instances starting to refuse connections as we seemed to have too many clients at once trying to open a connection
• we increased the number of open connections the SQL instances would allow
• we would still see connections being refused, albeit less
• to reduce the pressure on SQL the number of mediawiki replicas was reduced to 2 again, also memory "requests" were increased to make sure memory pressure on the node would not kill the pod
• the system ran slightly more stable, but once CPU usage got higher we'd still see memory usage skyrocketing and pods getting killed
• we increased the CPU limits and requests further (https://github.com/wmde/wbaas-deploy/pull/1248)
• now that CPU will never be close to being saturated, memory usage is at bay and pods do not crash anymore, even during high load