Page MenuHomePhabricator
Create Task
Maniphest T323557

Let HAProxy handle port 80
Closed, ResolvedPublic
Actions

Description

Analogue to the deprecated T254235, HAProxy should handle port 80 --> 443 redirection

Current state: the cache hosts on the following datacenters have been correctly updated (HAProxy and Varnish configurations) via the dedicated cookbook to swap port 80 management from Varnish to HAProxy.

  • drmrs (text and upload)
  • esams (text and upload)
  • codfw (text and upload)
  • eqiad (text and upload)

The Varnish and HAProxy configuration on the following datacenters has been updated previously without cookbook:

  • eqsin (text and upload)
  • ulsfo (text and upload)

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Vgutierrez triaged this task as Medium priority.Nov 22 2022, 10:11 AM
Vgutierrez created this task.
Vgutierrez updated the task description. (Show Details)
Maintenance_bot added a project: SRE.Nov 22 2022, 10:29 AM
BBlack mentioned this in T324336: Replace edge cache conftool entries 'varnish-fe' and 'ats-tls' with singular 'cdn'.Dec 2 2022, 2:52 PM
BCornwall changed the task status from Open to Stalled.Feb 22 2023, 6:05 PM
BCornwall moved this task from Backlog to Traffic team actively servicing on the Traffic board.
BCornwall subscribed.
BCornwall changed the task status from Stalled to In Progress.Apr 25 2023, 9:00 PM
BCornwall removed a project: SRE.May 1 2023, 4:46 PM
BCornwall merged a task: Restricted Task.May 2 2023, 8:23 PM
BCornwall added a subscriber: BTullis.
Fabfur subscribed.May 17 2023, 10:04 PM
Fabfur added a comment.EditedMay 22 2023, 3:16 PM

After talking with @Vgutierrez these are the steps the cookbook needs to implement:

  1. Depool the host
  2. Set profile::cache::varnish::frontend::enable_http_redirection: false (via netbox->hiera)
  3. Run puppet
  4. Restart varnish
  5. Set profile::cache::haproxy::http_redirection_port: 80 via netxbox->hiera
  6. Run puppet
  7. Repool the host
Fabfur added a subscriber: Volans.EditedMay 23 2023, 8:55 AM

The action plan is slightly changed thanks to the contribution of @Vgutierrez and @Volans .
Now the checklist is more like (on a per-DC basis):

  1. Disable puppet on all impacted hosts
  2. Merge the changes on hiera (no need to use netbox->hiera function anymore)

The next steps are on a per-host basis:

  1. Depool the host
  2. Stop Varnish
  3. Enable and run Puppet (varnish configuration will be updated, haproxy configuration will be updated and haproxy service reloaded)
  4. Start Varnish
  5. Test that haproxy is actually listening on port 80 and performs the correct redirect
  6. Repool the host
gerritbot added a comment.May 25 2023, 8:08 AM

Change 922844 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/cookbooks@master] Add a new cookbook that allows to run puppet configuration while restarting Varnish

https://gerrit.wikimedia.org/r/922844

gerritbot added a project: Patch-For-Review.May 25 2023, 8:08 AM
gerritbot added a comment.May 30 2023, 7:05 AM

Change 922844 merged by Vgutierrez:

[operations/cookbooks@master] SRE: Add a new cookbook that allows to run puppet configuration while restarting Varnish

https://gerrit.wikimedia.org/r/922844

Maintenance_bot removed a project: Patch-For-Review.May 30 2023, 7:10 AM
gerritbot added a comment.May 30 2023, 9:11 AM

Change 924444 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache::upload: Add hieradata to switch HTTPS redirection from Varnish to HAProxy

https://gerrit.wikimedia.org/r/924444

gerritbot added a project: Patch-For-Review.May 30 2023, 9:11 AM
Fabfur added a comment.May 30 2023, 12:37 PM

As the cookbook is ready for testing we can try to merge the hieradata and do the puppet disable on the codfw hosts.

gerritbot added a comment.May 30 2023, 2:21 PM

Change 924527 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/cookbooks@master] run-puppet-restart-varnish: fix _custom_action signature

https://gerrit.wikimedia.org/r/924527

gerritbot added a comment.May 30 2023, 2:36 PM

Change 924527 merged by jenkins-bot:

[operations/cookbooks@master] run-puppet-restart-varnish: fix _custom_action signature

https://gerrit.wikimedia.org/r/924527

gerritbot added a comment.May 30 2023, 6:01 PM

Change 924590 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/cookbooks@master] run-puppet-restart-varnish: Add dry_run support to check function

https://gerrit.wikimedia.org/r/924590

gerritbot added a comment.May 31 2023, 8:24 AM

Change 924590 merged by jenkins-bot:

[operations/cookbooks@master] run-puppet-restart-varnish: Add dry_run support to check function

https://gerrit.wikimedia.org/r/924590

gerritbot added a comment.May 31 2023, 8:56 AM

Change 924444 merged by Fabfur:

[operations/puppet@production] cache::upload: Switch HTTPS redirection from Varnish to HAProxy only on cp2042

https://gerrit.wikimedia.org/r/924444

Maintenance_bot removed a project: Patch-For-Review.May 31 2023, 9:11 AM
gerritbot added a comment.May 31 2023, 9:18 AM

Change 924894 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: Applying port 80 redirection on upload cluster in codfw

https://gerrit.wikimedia.org/r/924894

gerritbot added a project: Patch-For-Review.May 31 2023, 9:18 AM
gerritbot added a comment.May 31 2023, 9:57 AM

Change 924894 merged by Fabfur:

[operations/puppet@production] hiera: Swap port 80 from varnish to haproxy on codfw upload cluster

https://gerrit.wikimedia.org/r/924894

Maintenance_bot removed a project: Patch-For-Review.May 31 2023, 10:10 AM
gerritbot added a comment.May 31 2023, 12:46 PM

Change 924919 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Swap port 80 from varnish to haproxy on codfw@text

https://gerrit.wikimedia.org/r/924919

gerritbot added a project: Patch-For-Review.May 31 2023, 12:46 PM
gerritbot added a comment.May 31 2023, 1:23 PM

Change 924919 merged by Vgutierrez:

[operations/puppet@production] hiera: Swap port 80 from varnish to haproxy on text@codfw

https://gerrit.wikimedia.org/r/924919

Maintenance_bot removed a project: Patch-For-Review.May 31 2023, 1:30 PM
Stashbot mentioned this in T337247: ManagementSSHDown.May 31 2023, 4:13 PM

Mentioned in SAL (#wikimedia-operations) [2023-05-31T16:13:42Z] <vgutierrez> repool cp2035 - T337247 T323557

gerritbot added a comment.Jun 1 2023, 1:39 PM

Change 925779 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: Swap port 80 from varnish to haproxy on all drmrs clusters

https://gerrit.wikimedia.org/r/925779

gerritbot added a project: Patch-For-Review.Jun 1 2023, 1:39 PM
Stashbot added a comment.Jun 1 2023, 2:14 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-01T14:14:12Z] <fabfur> Disabled puppet on A:cp-drmrs for T323557

gerritbot added a comment.Jun 1 2023, 2:17 PM

Change 925779 merged by Fabfur:

[operations/puppet@production] hiera: Swap port 80 from varnish to haproxy on drmrs caching clusters

https://gerrit.wikimedia.org/r/925779

Maintenance_bot removed a project: Patch-For-Review.Jun 1 2023, 2:29 PM
Vgutierrez mentioned this in T337951: check_puppetrun fails to run under certain conditions.Jun 1 2023, 3:23 PM
gerritbot added a comment.Jun 6 2023, 7:36 AM

Change 927580 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: Swap port 80 from varnish to haproxy on drmrs upload cluster

https://gerrit.wikimedia.org/r/927580

gerritbot added a project: Patch-For-Review.Jun 6 2023, 7:36 AM
gerritbot added a comment.Jun 6 2023, 7:39 AM

Change 927580 merged by Fabfur:

[operations/puppet@production] hiera: Swap port 80 from varnish to haproxy on drmrs upload cluster

https://gerrit.wikimedia.org/r/927580

Maintenance_bot removed a project: Patch-For-Review.Jun 6 2023, 8:11 AM
gerritbot added a comment.Jun 6 2023, 9:15 AM

Change 927591 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: Swap port 80 from varnish to haproxy on drmrs text cluster

https://gerrit.wikimedia.org/r/927591

gerritbot added a project: Patch-For-Review.Jun 6 2023, 9:15 AM
gerritbot added a comment.Jun 6 2023, 9:25 AM

Change 927591 merged by Fabfur:

[operations/puppet@production] hiera: Swap port 80 from varnish to haproxy on drmrs text cluster

https://gerrit.wikimedia.org/r/927591

Maintenance_bot removed a project: Patch-For-Review.Jun 6 2023, 9:30 AM
gerritbot added a comment.Jun 6 2023, 2:29 PM

Change 927715 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: Swap port 80 from varnish to haproxy in eqiad

https://gerrit.wikimedia.org/r/927715

gerritbot added a project: Patch-For-Review.Jun 6 2023, 2:29 PM
Fabfur updated the task description. (Show Details)Jun 7 2023, 8:42 AM
Fabfur updated the task description. (Show Details)Jun 7 2023, 8:48 AM
Fabfur updated the task description. (Show Details)
Fabfur updated the task description. (Show Details)
gerritbot added a comment.Jun 7 2023, 8:52 AM

Change 927715 merged by Fabfur:

[operations/puppet@production] hiera: Swap port 80 from varnish to haproxy in eqiad

https://gerrit.wikimedia.org/r/927715

Maintenance_bot removed a project: Patch-For-Review.Jun 7 2023, 9:11 AM
Fabfur updated the task description. (Show Details)Jun 7 2023, 2:43 PM
Vgutierrez added a comment.Jun 7 2023, 2:46 PM
vgutierrez@cumin1001:~$ sudo -i cumin A:cp "ss --listen -t -p '( sport = :http )'  |grep haproxy |wc -l"
96 hosts will be targeted:
cp[2027-2042].codfw.wmnet,cp[6001-6016].drmrs.wmnet,cp[1075-1090].eqiad.wmnet,cp[5017-5032].eqsin.wmnet,cp[3050-3065].esams.wmnet,cp[4037-4052].ulsfo.wmnet
OK to proceed on 96 hosts? Enter the number of affected hosts to confirm or "q" to quit: 96
===== NODE GROUP =====
(96) cp[2027-2042].codfw.wmnet,cp[6001-6016].drmrs.wmnet,cp[1075-1090].eqiad.wmnet,cp[5017-5032].eqsin.wmnet,cp[3050-3065].esams.wmnet,cp[4037-4052].ulsfo.wmnet
----- OUTPUT of 'ss --listen -t -...p haproxy |wc -l' -----
2

thanks @Fabfur

Fabfur closed this task as Resolved.Jun 7 2023, 2:47 PM

All cp* hosts are now updated with HAProxy listening on port 80

taavi subscribed.Jun 8 2023, 4:46 PM

T338481: Confusing error message when making plaintext HTTP POST requests to Wikimedia sites seems to have been caused by this change.

gerritbot added a comment.Jun 14 2023, 8:51 AM

Change 929989 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: Consolidate http redirection directive across all DCs

https://gerrit.wikimedia.org/r/929989

gerritbot added a project: Patch-For-Review.Jun 14 2023, 8:51 AM
gerritbot added a comment.Jun 20 2023, 11:02 AM

Change 929989 merged by Fabfur:

[operations/puppet@production] hiera: Consolidate http redirection directive across all DCs

https://gerrit.wikimedia.org/r/929989

Maintenance_bot removed a project: Patch-For-Review.Jun 20 2023, 11:11 AM
Vgutierrez closed subtask T339898: port 80 paging on scheduled single host maintenance in text@esams as Resolved.Jun 22 2023, 9:17 AM
Vgutierrez added a subtask: T340097: Webrequests live data shows traffic without TLS on varnish for upload.w.o.
Vgutierrez removed a subtask: T340097: Webrequests live data shows traffic without TLS on varnish for upload.w.o.Jun 22 2023, 9:24 AM
gerritbot added a comment.Jun 28 2023, 10:54 AM

Change 933881 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache: Setting port 80 as default redirection port in haproxy

https://gerrit.wikimedia.org/r/933881

gerritbot added a project: Patch-For-Review.Jun 28 2023, 10:54 AM
gerritbot added a comment.Jun 28 2023, 1:16 PM

Change 933881 merged by Fabfur:

[operations/puppet@production] cache: Setting port 80 as default redirection port in haproxy

https://gerrit.wikimedia.org/r/933881

Maintenance_bot removed a project: Patch-For-Review.Jun 28 2023, 1:31 PM
gerritbot added a comment.Jun 29 2023, 1:13 PM

Change 934328 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] varnish: Remove http/https redirection

https://gerrit.wikimedia.org/r/934328

gerritbot added a project: Patch-For-Review.Jun 29 2023, 1:13 PM
gerritbot added a comment.Jul 3 2023, 11:05 AM

Change 934328 merged by Fabfur:

[operations/puppet@production] varnish: Remove http/https redirection

https://gerrit.wikimedia.org/r/934328

Maintenance_bot removed a project: Patch-For-Review.Jul 3 2023, 11:10 AM
Fabfur added a comment.Jul 18 2023, 10:14 AM

The HAProxy configuration on all DCs has been updated to apply silent-drop to abusive clients hitting port 80, as been already done for port 443.

To check (eg. from cumin) if HAProxy is "silent-dropping" connections:

For port 443:
sudo cumin --ignore-exit-codes A:cp 'journalctl -u haproxy --since=-1h | grep silent-drop_for'

For port 80:
sudo cumin --ignore-exit-codes A:cp 'journalctl -u haproxy --since=-1h | grep silent-drop_port80_for'

Fabfur mentioned this in T351117: Move analytics log from Varnish to HAProxy.Nov 15 2023, 8:48 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits