Change 730923 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/puppet@production] [WIP] mediawiki: Disable mod_unique_id

https://gerrit.wikimedia.org/r/730923

I couldn't figure out exactly what was enabling mod_unique_id in the first place. On mw1320, I see:

legoktm@mw1320:/etc/apache2$ ls -l mods-enabled/
...
lrwxrwxrwx 1 root root 32 Feb 19  2021 unique_id.load -> ../mods-available/unique_id.load

at roughly the timestamp of reimage, so it either got enabled via the package or puppet. But when I plain install apache2 in a container, it does not get enabled by default.

I don't see it in puppet either, grepping for unique_id (case-insensitive) turns up no results. And the patch I submitted passed PCC, which I would've expected some conflict if it was coming from puppet.

In T253675#7430025, @Legoktm wrote:

I couldn't figure out exactly what was enabling mod_unique_id in the first place.

@Legoktm I made some tests and found it out. It is installed once you install the libapache2-mod-security2 package. Doing this also gets you the unique_id module. (Tested on my local computer, it's not there first but it is after installing that package).

And we have package { 'libapache2-mod-security2': in profile::mediawiki::httpd.

grep Depends /etc/apache2/mods-available/security2.load 
# Depends: unique_id

Change 730923 abandoned by Legoktm:

[operations/puppet@production] [WIP] mediawiki: Disable mod_unique_id

Reason:

Needed by mod_security2

https://gerrit.wikimedia.org/r/730923

Oooh, I forgot to look for that, thanks! I think that would necessitate declining this task, unless mod_security2 is also not really needed. AFAICT we currently use it for:

# Make the Server response header equal to the host's FQDN.
<IfModule security2_module>
    ServerTokens Full
    SecServerSignature "<%= scope['::fqdn'] %>"
</IfModule>

Which could just be a standard response header addition? Not sure if it's used elsewhere.

This was added in T132599 (https://phabricator.wikimedia.org/rOPUP646adf13b514cd4f380055a3bee9a7f4955310a3) where Ori described it as "give us the ability to ban cache objects that were generated by codfw app servers, in case codfw app servers produce mangled responses after the switch over". So the question would be if we still need that ability.

It seems that is indeed the only use, and there's a couple of workaround and bugs around this module that may be possible to remove if it too were removed.

https://codesearch.wmcloud.org/operations/?q=mod.sec&i=nope&files=&excludeFiles=&repos=

# modules/profile/manifests/mediawiki/httpd.pp
# Set the Server response header to be equal to the app server FQDN.
package { 'libapache2-mod-security2':
        ensure => present

# modules/profile/manifests/mediawiki/httpd.pp
# Starting with stretch libapache2-mod-security2 includes the following
# in /etc/apache2/mods-enabled/security2.conf: […]
# Once we're running a version of the patch proposed in Apache bugzilla, this
# workaround can be removed

# /modules/profile/manifests/mediawiki/maintenance.pp
# Installing libapache2-mod-security2 without also installing modsecurity-crs
# leads to a syntax error due to a bug in the former package […]
package { [ 'libapache2-mod-security2', 'modsecurity-crs']:

It seemed strange to me that we use mod_security here since it seems the handful of things that mod_security does, we immediately undo in our conf file, and the only use case we have for it (display FQDN) is something we do in our conf and not by mod_security.

For example, the apache2 default is ServerTokens Full, which mod_security changes to ServerTokens OS, and then our code sets it back to ServerTokens Full.

As I understand it, the reason we used mod_security for this is that Apache does not allow unsetting or changing of the Server header by any other means. If we still want a way to use Varnish bans by backend datacenter, I suppose we could emit it through a different header like X-Apache-Hostname (which we can then strip out in ATS, as we do with various other headers we don't need externally).

I do note that through the transition from Varnish to ATS, we no longer reliably get Apache's "Server" header at the edge. Quite often it is now observed as Server: ATS. I don't know if that's done only by ats-tls or whether ats-be does it (and thus Varnish doesn't see the Apache one).

In T253675#7430192, @Krinkle wrote:

As I understand it, the reason we used mod_security for this is that Apache does not allow unsetting or changing of the Server header by any other means. If we still want a way to use Varnish bans by backend datacenter, I suppose we could emit it through a different header like X-Apache-Hostname (which we can then strip out in ATS, as we do with various other headers we don't need externally).

@ori, do you remember why you wanted to overload "Server" for this rather than adding a new header?