Page MenuHomePhabricator

[EPIC][Infra] Move Wikibase and WikibaseLexeme Git submodules to suitable Git host
Open, Needs TriagePublic

Description

Context

The Wikibase and WikibaseLexeme extension Git repositories have several submodules: some libraries and the new (mobile) termbox in Wikibase, and the new Special:NewLexeme page in WikibaseLexeme. Git repository options for them include:

  • GitHub: Currently the “canonical” repository for several of them (i.e. where actual development happens, if it happens); however, due to production network restrictions, GitHub cannot be used as the upstream URL of the submodules (T301273, I7b1e3cbafc)
  • Wikimedia Phabricator (Diffusion): Currently used as a mirror for the four repositories that live on GitHub, and registered in .gitmodules for them to work around the network restrictions mentioned above; however, access to Phabricator from cloud networks is severely rate-limited, so it is effectively not possible to clone these repositories from within GitHub Actions
  • Wikimedia Gerrit: Currently the “canonical” repository and .gitmodules URL for the three remaining submodules, but enforces a different development model
  • Wikimedia GitLab: Currently not used for any submodules, but allows a mostly GitHub-like development model without the network restrictions affecting GitHub and Phabricator

Main Objectives

  • Move the submodules for Wikibase and WikibaseLexeme from Phabricator, to GitLab. This includes
    • [Wikibase] view/lib/wikibase-serialization
    • [Wikibase] view/lib/wikibase-data-values
    • [Wikibase] view/lib/wikibase-data-model
    • [WikibaseLexeme] resources/special/new-lexeme

Mitigated Risks

  • Development slowdowns and stalls due to Diffusion's rate limiting
  • High context switching due to different tool use

Potential Tasks

...

Notes

We have several options to achieve our objective:

  1. Create the GitLab repositories and manually keep them up to date (i.e. push to them whenever we push to the original GitHub repositories, or at least whenever we want to update the submodule commit in Wikibase / WikibaseLexeme) while keeping development on GitHub
  2. Have the GitLab repositories automatically mirror the GitHub repositories, by having GitHub push to them from GitHub actions (having GitLab pull from GitHub automatically is sadly no longer supported – moved to GitLab Premium in 13.9)
  3. Actually move development of the repositories to GitLab and treat GitLab as the “canonical” source

I think these options are not mutually exclusive – in fact, it’s probably easiest to start with 1 and then optionally upgrade to 2 and/or 3 later if we want to.

Open question: Which GitLab namespace should we use? CC @thcipriani or @LSobanski from our earlier discussion – is there a recommended namespace for GitLab repositories that will be used in the production MediaWiki environment?

Event Timeline

One case that needs testing is whether GitHub pulling from GitLab could be influenced by the work done in T366882: implement anti-abuse features for GitLab (Move GitLab behind the CDN). Do we know how many concurrent connections GitHub Actions create?

I don’t think we have any specific access numbers. If the limits are reasonably high, I think we’ll just have to hope for the best – it doesn’t sound like we have other options anyway.

ItamarWMDE renamed this task from Move Wikibase and WikibaseLexeme Git submodules to suitable Git host to [Infra][EPIC] Move Wikibase and WikibaseLexeme Git submodules to suitable Git host.Thu, Sep 26, 1:28 PM
ItamarWMDE updated the task description. (Show Details)
ItamarWMDE added subscribers: WMDE-leszek, ItamarWMDE.

@Lucas_Werkmeister_WMDE I've rewritten this ticket to match our EPIC format, please feel free to update any missing information or correct any mistakes.

@WMDE-leszek I saw you sent a message regarding this topic, are your plans impacted by this, are they covered by this epic?

ItamarWMDE renamed this task from [Infra][EPIC] Move Wikibase and WikibaseLexeme Git submodules to suitable Git host to [EPIC][Infra] Move Wikibase and WikibaseLexeme Git submodules to suitable Git host.Thu, Sep 26, 1:31 PM
ItamarWMDE moved this task from Incoming to [DOT] Epics Stalled on the wmde-wikidata-tech board.