Page MenuHomePhabricator
Create Task
Maniphest T210704

Migrate node-based services in production to node10
Open, MediumPublic
Actions

Description

Initial task | T290750: Migrate node-based services in production to node12

The majority of our node-based services is currently running Node 6 and that branch is EOLed 2019-04-01: https://nodejs.org/en/about/releases/

I can backport security fixes for a while, but that's not ideal and we should plan the migration of node-based services to 10 (which also implies stretch as node10 has hard requirements on libraries only in stretch).

Services currently using nodejs:

node10 debs for stretch are available in the repository component "component/node10" , see T203239 for further details. After https://gerrit.wikimedia.org/r/477475 service::node deploys the stretch nodejs10 component with the parameter use_nodejs10 set to true.

Details

SubjectRepoBranchLines /-
Docker: Drop node10, unusedintegration/configmaster 0 -92
maps: upgrade to nodejs10operations/puppetproduction 1 -1
service::node: add the 'use_nodejs10' parameteroperations/puppetproduction 34 -4
turnilo: fix dependency cycle removing require_packageoperations/puppetproduction 4 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
MSantos mentioned this in T213077: Migrate Kartotherian to node-mapnik >v4.2.1 to support libmapnik >3.1.x .Jan 7 2019, 1:53 PM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:46 PM
MSantos mentioned this in T215852: Migrate Kartotherian/Tilerator to Node 10.Feb 12 2019, 12:49 AM
gerritbot added a comment.Feb 18 2019, 3:45 PM

Change 477475 merged by Elukey:
[operations/puppet@production] service::node: add the 'use_nodejs10' parameter

https://gerrit.wikimedia.org/r/477475

bd808 added a subtask: T213077: Migrate Kartotherian to node-mapnik >v4.2.1 to support libmapnik >3.1.x .Feb 18 2019, 4:19 PM
jijiki updated the task description. (Show Details)Feb 19 2019, 11:27 AM
jijiki added a project: serviceops.
elukey updated the task description. (Show Details)Feb 19 2019, 11:56 AM
hashar unsubscribed.Feb 20 2019, 1:40 PM
Tgr mentioned this in T217114: Migrate Proton to k8s and nodejs 10.Feb 26 2019, 4:17 AM
Tgr added a subtask: T217114: Migrate Proton to k8s and nodejs 10.
Ottomata added a subtask: T218238: Vagrant initial provision fails on NodeJS version mismatch.Mar 14 2019, 1:42 PM
• Nuria closed subtask T210706: Move AQS to nodejs 10 as Resolved.Mar 18 2019, 4:16 PM
Kanashimi subscribed.Mar 19 2019, 10:16 PM
• Nuria closed subtask T218238: Vagrant initial provision fails on NodeJS version mismatch as Resolved.Apr 19 2019, 1:38 PM
gerritbot added a comment.May 8 2019, 1:37 PM

Change 508822 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: upgrade to nodejs10

https://gerrit.wikimedia.org/r/508822

Stashbot added a comment.May 8 2019, 2:03 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-08T14:03:26Z] <gehel> starting upgrade to nodejs 10 for maps - T210704

gerritbot added a comment.May 8 2019, 2:05 PM

Change 508822 merged by Gehel:
[operations/puppet@production] maps: upgrade to nodejs10

https://gerrit.wikimedia.org/r/508822

Stashbot added a comment.May 8 2019, 7:26 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-08T19:26:27Z] <gehel> continue upgrade to nodejs 10 for maps - T210704

Stashbot added a comment.May 8 2019, 8:10 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-08T20:10:42Z] <gehel> upgrade to nodejs 10 for maps completed - T210704

Maintenance_bot removed a project: Patch-For-Review.May 22 2019, 3:45 PM
Jdforrester-WMF mentioned this in T211784: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects.Jun 12 2019, 5:42 PM
KartikMistry subscribed.Jun 14 2019, 5:38 AM
KartikMistry added a comment.Jun 19 2019, 6:59 AM

We recently tried to upgrade to nodejs10 for cxserver but it seems zlib 1.2.11 is required.

Example error: MT processing error for: en > qqq. Error: invalid distance too far back at Zlib.zlibOnError [as onerror]

See: https://github.com/nodejs/node/issues/22839

MoritzMuehlenhoff added a comment.Jun 19 2019, 7:02 AM

Are you using component/node10? This should be fixed already, see https://phabricator.wikimedia.org/T215562#5066711 and followups.

Joe subscribed.Jun 19 2019, 7:24 AM
In T210704#5267695, @MoritzMuehlenhoff wrote:

Are you using component/node10? This should be fixed already, see https://phabricator.wikimedia.org/T215562#5066711 and followups.

given they're using our base images, I guess we have to update the nodejs base images to use it.

Joe added a comment.Jun 19 2019, 7:28 AM
In T210704#5267739, @Joe wrote:
In T210704#5267695, @MoritzMuehlenhoff wrote:

Are you using component/node10? This should be fixed already, see https://phabricator.wikimedia.org/T215562#5066711 and followups.

given they're using our base images, I guess we have to update the nodejs base images to use it.

To correct myself: we already use that component. I'm nonetheless creating a new version of the base image. @KartikMistry what base image are you using for your project?

KartikMistry added a comment.Jun 19 2019, 7:39 AM
In T210704#5267750, @Joe wrote:

To correct myself: we already use that component. I'm nonetheless creating a new version of the base image. @KartikMistry what base image are you using for your project?

docker-registry.wikimedia.org/nodejs10-slim and docker-registry.wikimedia.org/nodejs10-devel

KartikMistry added a comment.Jun 20 2019, 4:54 AM

@Joe @MoritzMuehlenhoff You can also check, https://phabricator.wikimedia.org/diffusion/GCXS/browse/master/.pipeline/blubber.yaml if something is wrong there.

Joe added a comment.Jun 20 2019, 5:58 AM
In T210704#5267755, @KartikMistry wrote:
In T210704#5267750, @Joe wrote:

To correct myself: we already use that component. I'm nonetheless creating a new version of the base image. @KartikMistry what base image are you using for your project?

docker-registry.wikimedia.org/nodejs10-slim and docker-registry.wikimedia.org/nodejs10-devel

You have the problem with both? In theory the nodejs10-devel image should use the official packages from nodesource, so it shouldn't be affected, unless I'm missing something.

akosiaris added a comment.Jun 20 2019, 7:45 AM
In T210704#5270288, @Joe wrote:
In T210704#5267755, @KartikMistry wrote:
In T210704#5267750, @Joe wrote:

To correct myself: we already use that component. I'm nonetheless creating a new version of the base image. @KartikMistry what base image are you using for your project?

docker-registry.wikimedia.org/nodejs10-slim and docker-registry.wikimedia.org/nodejs10-devel

You have the problem with both? In theory the nodejs10-devel image should use the official packages from nodesource, so it shouldn't be affected, unless I'm missing something.

I don't think they 've had to interface directly with the nodejs10-devel image as the tests pass successfully (so probably no problem there).

akosiaris added a comment.Jun 20 2019, 7:56 AM
In T210704#5270230, @KartikMistry wrote:

@Joe @MoritzMuehlenhoff You can also check, https://phabricator.wikimedia.org/diffusion/GCXS/browse/master/.pipeline/blubber.yaml if something is wrong there.

Could you please describe how to reproduce this? It seems we know exactly what it is but just to make sure when we are build the new images.

KartikMistry added a comment.Jun 20 2019, 9:05 AM
In T210704#5270430, @akosiaris wrote:
In T210704#5270230, @KartikMistry wrote:

@Joe @MoritzMuehlenhoff You can also check, https://phabricator.wikimedia.org/diffusion/GCXS/browse/master/.pipeline/blubber.yaml if something is wrong there.

Could you please describe how to reproduce this? It seems we know exactly what it is but just to make sure when we are build the new images.

I was able to reproduce error we saw in Production using end point tests manually with docker image tag (2019-06-18-094614-production).

KartikMistry added a comment.Jun 24 2019, 4:07 AM
In T210704#5270565, @KartikMistry wrote:

I was able to reproduce error we saw in Production using end point tests manually with docker image tag (2019-06-18-094614-production).

Let me know if any further information needed.

Joe closed subtask T226346: Update nodejs10 image to use the latest version of the package as Resolved.Jun 24 2019, 6:31 AM

@KartikMistry if we trigger a rebuild of the production container, it should now use the newer nodejs10-slim image and work as expected. Can you please confirm?

KartikMistry added a comment.Jun 25 2019, 4:50 AM
In T210704#5277211, @Joe wrote:

@KartikMistry if we trigger a rebuild of the production container, it should now use the newer nodejs10-slim image and work as expected. Can you please confirm?

This looks good. Deployed in Production.

KartikMistry closed subtask T226074: Update cxserver to nodejs10 as Resolved.Jul 2 2019, 5:26 AM
MoritzMuehlenhoff mentioned this in T211881: graphoid: Code stewardship request.Jul 17 2019, 7:12 AM
• Pchelolo edited projects, added Platform Engineering (Needs Cleaning - Services Operations); removed Platform Team Legacy (Later), Services (next).Jul 17 2019, 5:00 PM
MSantos updated the task description. (Show Details)Jul 18 2019, 9:22 PM
MoritzMuehlenhoff updated the task description. (Show Details)Aug 16 2019, 7:50 AM
Jdlrobson added a subtask: T230841: Migrate documentation generation to Node 10.15.2 from node 6.11.0.Aug 22 2019, 2:17 PM
Jdforrester-WMF removed a subtask: T230841: Migrate documentation generation to Node 10.15.2 from node 6.11.0.Aug 26 2019, 6:31 PM
elukey updated the task description. (Show Details)Sep 26 2019, 7:29 AM
• Mholloway closed subtask T217114: Migrate Proton to k8s and nodejs 10 as Resolved.Sep 30 2019, 3:55 PM
• Mholloway reopened subtask T217114: Migrate Proton to k8s and nodejs 10 as Open.Sep 30 2019, 4:02 PM
Jdforrester-WMF updated the task description. (Show Details)Sep 30 2019, 5:05 PM
• WDoranWMF edited projects, added Core Platform Team Initiatives (Containerise Services); removed Platform Engineering (Needs Cleaning - Services Operations).Nov 1 2019, 9:49 PM
Tgr mentioned this in T217113: MediaWiki-Vagrant should use the same Node.js version as Wikimedia production.Nov 22 2019, 11:11 PM
MSantos closed subtask T217114: Migrate Proton to k8s and nodejs 10 as Resolved.Jun 23 2020, 3:33 PM
• Mholloway reopened subtask T217114: Migrate Proton to k8s and nodejs 10 as Open.Jun 29 2020, 1:40 PM
Jdforrester-WMF closed subtask T217114: Migrate Proton to k8s and nodejs 10 as Resolved.Jul 10 2020, 10:12 AM
akosiaris closed subtask T218733: Migrate mobileapps to k8s and node 10 as Resolved.Aug 5 2020, 1:48 PM
Dzahn subscribed.Aug 11 2020, 8:53 PM

Checking the box for phabricator/aphlict. aphlict is now running on a dedicated VM, aphlict1001, on buster and nodejs 10. The phabricator server phab1001 is also on buster and nodejs10. (which we could maybe remove now).

Dzahn updated the task description. (Show Details)Aug 11 2020, 8:53 PM

Also checking the box for etherpad. That is also on buster and nodejs10 meanwhile. Upgraded by Alex Kosiaris.

Dzahn updated the task description. (Show Details)Aug 11 2020, 8:55 PM
jijiki moved this task from Incoming 🐫 to 🔦Unused2 on the serviceops board.Aug 17 2020, 11:48 PM
jijiki moved this task from 🔦Unused2 to 🗄 Projects on the serviceops board.Aug 18 2020, 6:56 AM
Tgr mentioned this in T236975: Vagrant eventbus role can't be installed because of node_version.Aug 18 2020, 12:32 PM
• Pchelolo mentioned this in T263489: AQS 2.0.Sep 21 2020, 6:18 PM
Dzahn added a comment.Oct 20 2020, 8:35 PM

parsoid: WIP in https://gerrit.wikimedia.org/r/c/operations/puppet/ /634383 / T257906

hashar mentioned this in T267891: Create Debian packages for Node.js 14 upgrade.Nov 16 2020, 9:06 AM
• Mholloway updated the task description. (Show Details)Mar 5 2021, 9:10 PM
gerritbot added a comment.Sep 10 2021, 2:24 AM

Change 720156 had a related patch set uploaded (by Jforrester; author: Jforrester):

[integration/config@master] Docker: Drop node10, unused

https://gerrit.wikimedia.org/r/720156

gerritbot added a project: Patch-For-Review.Sep 10 2021, 2:24 AM
Jdforrester-WMF mentioned this in T290750: Migrate node-based services in production to node12.Sep 10 2021, 4:52 PM
Jdforrester-WMF updated the task description. (Show Details)Sep 10 2021, 5:00 PM
Jdforrester-WMF updated the task description. (Show Details)
Jdforrester-WMF updated the task description. (Show Details)
Jdforrester-WMF subscribed.

Has Thumbor been upgraded, or is this waiting on T216815: Upgrade Thumbor to Buster?

AntiCompositeNumber subscribed.Sep 10 2021, 5:10 PM
akosiaris added a comment.Sep 13 2021, 1:58 PM
In T210704#7345278, @Jdforrester-WMF wrote:

Has Thumbor been upgraded, or is this waiting on T216815: Upgrade Thumbor to Buster?

Thumbor has not been upgraded yet, but also thumbor isn't nodejs. It's python.

akosiaris added a comment.Sep 13 2021, 2:00 PM

Ah, 3d2png is also shipped with thumbor and that one is indeed nodejs. I now get the question, sorry about that. Anyway, It's still not buster unfortunately.

MoritzMuehlenhoff added a comment.Sep 13 2021, 2:12 PM

Not sure why restbase is ticked off, though? The restbase hosts in production currently run nodejs 6.11 still.

Jdforrester-WMF updated the task description. (Show Details)Sep 13 2021, 6:14 PM
Jdforrester-WMF added a comment.Sep 13 2021, 6:16 PM
In T210704#7348478, @MoritzMuehlenhoff wrote:

Not sure why restbase is ticked off, though? The restbase hosts in production currently run nodejs 6.11 still.

Oh wow, really? I thought that CI switched over, but checking e.g. https://github.com/wikimedia/restbase/pull/1294/checks shows we're still running that. Ouch.

MoritzMuehlenhoff updated the task description. (Show Details)Sep 14 2021, 12:38 PM
MoritzMuehlenhoff updated the task description. (Show Details)
awight removed a subtask: T213077: Migrate Kartotherian to node-mapnik >v4.2.1 to support libmapnik >3.1.x .Feb 10 2022, 3:12 PM
gerritbot added a comment.Feb 21 2022, 5:41 PM

Change 720156 merged by jenkins-bot:

[integration/config@master] Docker: Drop node10, unused

https://gerrit.wikimedia.org/r/720156

akosiaris edited projects, added serviceops-radar; removed serviceops.Sep 29 2022, 12:46 PM
Aklapper removed subscribers: • Fjalapeno, • mobrovac.Aug 14 2023, 11:38 AM

Looking at the remaining items in this task description,

If the answer to both questions is "yes", can this task be resolved?

Jdforrester-WMF added a comment.Aug 18 2023, 6:13 AM
In T210704#9089606, @Aklapper wrote:

Not superseded, that's apparently now a sub-task of this, resolving the 3d2png part. Will mark accordingly.

  • Is Restbase superseded by T290750?

No, that's the follow-on task after this one is done.

Jdforrester-WMF added a subtask: T225678: Migrate 3d2png to k8s.Aug 18 2023, 6:14 AM
hnowlan closed subtask T225678: Migrate 3d2png to k8s as Resolved.Oct 2 2023, 9:36 AM
Jdforrester-WMF updated the task description. (Show Details)Oct 2 2023, 1:10 PM
Dzahn unsubscribed.Nov 2 2023, 10:58 PM
Jdforrester-WMF mentioned this in T290756: Migrate restbase production service to node12.Aug 13 2024, 1:21 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits