Background
mediawiki-extensions-Popups currently uses a compilation step (npm run build) to pull together front-end assets and process them before pointing ResourceLoader at them for serving.
Currently...
- Compiled front-end assets are generated by developers and added to the git source tree under [resources/dist](https://github.com/wikimedia/mediawiki-extensions-Popups/tree/master/resources/dist) on every patch that touches front-end sources.
- We run a script on the npm job that verifies that the compiled assets are up to date, and bails if they weren't properly committed.
This allows us to:
- Keep master updated with the deploy assets
- Ensure that latest sources will have the latest deploy assets on master (with the CI step)
Cons
Rebase hell
Because of how it is set up, every pending patch on gerrit that touches the frontend assets, needs to be rebased every time master is updated, to get rid of the merge conflict and regenerate the compiled assets on top of it.
This can get very annoying for pending changes as any change merged will force a rebase on all pending changes.
It's so annoying we made a bot to do this for us (see T167181 Create bot that automatically rebases and rebuilds patches to master)
Example:
- There are changes A, B and C posted on gerrit based off master. They touch front-end assets and as such generate assets into resources/dist with npm run build
- Reviewer goes and 2s patch A.
- Goes to review B, wants to 2, but because A is merged to master and changed the resources/dist files, B now Cannot merge.
- B and C now need to be rebased on top of master, run the build step, and re-submitted with updated compiled assets.
Cannot configure per wiki
When working on T173491 it was raised that because $wgScriptPath is configurable map files compiled from this process cannot be changed on a per wiki basis. A formal build step could potentially allow us to build on deploy and thus provide debugging on these wikis
What do we need
To not submit compiled assets with the patches, but keep the deploy branch (master currently) updated with the compiled assets.
Solutions considered
- Keep doing what we are doing
- Pros: Works, nothing to change
- Cons: Pretty painful if there is churn in the repo because of the merge conflicts
- Don't submit the compiled assets on every patch
- Manually build and release to the master branch
- Pros
- No merge conflicts because of stale compiled assets
- Control over the front-end source releases
- Cons
- Backend and front-end source could be out of sync on a release if assets not compiled appropriately
- Front-end not auto updated in staging and beta cluster environments
- Manual step, bound to fail because humans
- Make CI run the build step and merge compiled assets
- With the pros of what we are doing currently, but without the cons
- Pros
- Simpler workflow compared to the current one as there's no need to remember to add the compiled assets to the change.
- Keeps master updated with the deploy assets
- Ensure that latest sources will have the latest deploy assets on deploy branch (master)
- Cons
- Needs work with RelEng team to help us automate this
- Seems to be a dirty solution, having a separate deploy branch/repo seems to be preferred
- There are jobs that need change to learn how to build assets, like the selenium jobs, otherwise their results will be moot
- Needs work with RelEng team to help us automate this
Ideally after chatting, option 3. Make CI run the build step and merge compiled assets would be our preferred choice if RelEng deems it feasible and not problematic. We had an initial chat with @hashar about different ways this could be accomplished.
Questions
- Is this feasible? (amount of work to get it set up)
- How often do we run the job/script to generate assets and merge to the deploy branch (master)? (hourly? daily? after each commit?)
- Other repositories under wmf keep the deployable version of the project under a project/deploy branch. Is it worth it to have a separate deploy branch and sync it/submit compiled assets there? (Given this is a mediawiki extension) Or is it better to keep master as the deployable branch and push the assets there?
- Is this a good chance to define a general CI process to trigger a build step on projects? (for bundling frontend assets, pre-compiling templates, optimizing static assets like images, etc)