Move back to NPM

We've been wanting to move back from Yarn to NPM for a while now. Our infrastructure currently uses Yarn to install dependencies, and we are stuck in version 1.2, which is the latest 1.x version. Migrating to Yarn 2.x requires a lot of change to our infrastructure and some of the changes that Yarn 2.x introduces are "controversia" at the very least.

At the same time NPM has "grown" in capabilities, performance and reliability, to the point that it is now a viable alternative to Yarn. Having to use Yarn introduces additional software to the stack that makes some goals that we have harder to achieve:

Our Source Build migration of NPM packages is blocked on Source Build support for Yarn, which is not a priority for that team. This causes us to have painful workaround like checking-in versions of our JS build outputs into source control to support source build. Which inevitably cause conflicts among developers making changes EVERY SINGLE TIME.

Goals

• Unblock Source Build for NPM packages and dependencies.
• Avoid having to check-in JS build outputs into source control to support Source Build.
• Reduce the number of external dependencies in our infrastructure.
• Leverage modern NPM features like workspaces, overrides, etc.
• Unblock centralizing our NPM dependencies.
• Unblock centralizing our JavaScript build infrastructure configuration.
• Simplify dependency updates.

Implementation details

This change removes Yarn from our infrastructure completely and replaces it with NPM. There is a package.json in the top folder of the respository that defines the workspace. The workspace lists all the paths to packages that are part of the repository. In addition to that, the top package.json defines scripts that can be run from the root of the repository to install, build test, etc. all the packages in the repository.

Each package has its own package.json file that defines its dependencies and scripts. The scripts in the top package.json file are just wrappers around the scripts in the individual package.json files.

Most of the Yarn related code and scripts has been removed. We've still left the .npmproj files that were part of our custom infrastructure, but these are no longer used.

Developer workflow

"Restore dependencies"

To install the dependencies, you can run npm ci from the root of the repository. This will install all the dependencies for all the packages in the repository and automatically link them together. This normally takes very little time once you've have downloaded the packages onto your machine (10s for me).

"Install or update new dependencies"

Run npm install ... on whatever node you want to install the dependencies. The package-lock.json will get updated accordingly. For installing new dependencies you might need to authenticate yourself with the NPM VSTS feed (this was the same with yarn). To do that, simply run:

• npm install -g vsts-npm-auth
• vsts-npm-auth -config .npmrc -F which will walk you through the auth process and provision the required credentials.
  • At that point, you should be able to run npm install

Updating deps to fix Component Governance alerts

• Run npm audit to see the list of issues.
• Run npm audit fix to see if the issues can get addressed automatically.
• If that's not the case:
  • If possible, update to a newer version of the package that doesn't include the vulnerable dependency.
  • If that's not possible, at the appropriate node, add an entry to the "overrides" property to force the transitive dependency into a non-vulnerable version.
  • run npm install from the root.
  • run npm audit again to see if the issues have disappeared, if they haven't continue making changes (update more packages, etc).
  • Upgrading the main deps should always be the preferred route over using overrides. Overrides require that we periodically remove them to check if we can update the dependency later on.

From there, you can run npm run build to build all the packages in the repository. This will run the build script in each package. The build script is defined in the top package.json file and it just runs npm run build in each package of the workspace if it defines a build command. You can also navigate to a specific package and run npm run build to build just that package.

CI workflow

There is a project file in eng\Npm.Workspace.proj that drives all operations in the workspace, which in turn use idiomatic node/npm features to perform the operations:

• Restore: Runs npm ci in the root of the repository.
• Build: Runs npm run build in the root of the repository.
• Test: Runs npm run test in the root of the repository.
• Pack: Runs a script in eng\scripts\pack-workspace.js to version and pack all the packages in the repository that aren't marked as "private".

Open issues

As part of this chunk of work we have skiped some further cleanups we can do to limit the size of the changes:

• Remove eng/Build.targets as its not needed
• Remove console.log from src/Components/Web.JS/jest.config.js
• Remove all the npx stuff as its not needed.
• Remove all the .npmproj files as they are not needed.
• Reenable the HTTPS functional tests on SignalR.
• Update the dependencies in SignalR packages to use "*" instead of "^5.0.0" dev.
• Update docs, etc. on the repo.

Follow-up work

• Reengage with Source Build to get the JS bits building there and avoid having to check-in JS build outputs into source control.
• Start centralizing our dependencies and build infrastructure (SignalR, Blazor, etc.)
• Further cleanup of things that are no longer needed as part of the repository.
  • Yarn SDK, and anything related to NPM proj.
• Start centralizing our JS build dependencies.