Covered components depend on what WDME bundles and how. If Docker images are a distribution mechanism, security issues with e.g. MediaWiki should trigger a new build, since you're distributing it.

Where Docker compose instructions use certain dependencies, I think it's incumbent on WMDE to at least point people in the right direction for staying secure - and you may want to do more, to ensure people get the features you're working on. What exists now is only suitable for production if you a) know enough Docker to compose your own, and b) are subscribed to MediaWiki announcements so as to know when to rebuild. The docs don't tell you to do b), so you end up with people stuck on 1.34.0 (taking an active example from the Wikibase Registry), rather than going to 1.34.4 and then 1.35.1/2.

T266574 raised this issue for wikibase-docker - does it use "the regular mediawiki container images"? It seems so. Are they updated in a timely fashion? The situation may be improved from T197220, but it's not clear - it'd be good to have that briefly documented. But if MediaWiki's images update, do Wikibase's? It looks like this isn't the case (T275217); I tried downloading from 1.35-base and the changelog in the MediaWiki layer is from MediaWiki 1.35.1 - because the last update to master in the docker repo was on March 23 - even though security release 1.35.2 came out on the 8th.

I read that repo is being deprecated in favour of a "V1", but this should be fixed in that; and there should be a plan for getting people to update to "V1.1" or "V2", so that sites have secure versions of Wikibase and any bundled components; MediaWiki, WDQS, etc. Announcing releases in a suitable way for the distribution mechanism (and encouraging subscription to it) should be part of the process.

To potentially include in the task:

• Decide on the process of releasing and communicating security fixes to Wikibase and other WMDE maintained software
• Decide on the process of including security fixes of non-WMDE maintained software (e.g. Mediawiki, Elasticearch, Query Service backend) and publishing security fix releases of Wikibase suite in such cases.
• Document - for WMDE use - list of steps in the process of recognizing, fixing and releasing security updates to WMDE maintained software
• Provide an overview - for public use - of how security releases of Wikibase suite are going to be published
• Document - for WMDE use - list of communication channels to be be used to communicate upcoming, and published security releases
• Document - in Wikibase docs - a "recommended" communication channel used for communication on security-related updates (e.g. Wikibase mailing list), so that users know where to expect important updates

Timebox: 4hrs

An attempt at answering these AC:s

https://github.com/wmde/wikibase-release-pipeline/pull/169

One issue that strikes me from what's written is that WDQS has (at least in recent months, I think?) not been updated in the docker package - at a guess, because of ongoing work on streaming updater, or because there were no specific new features that Wikibase was relying on. The latest version in use (0.3.40) is from early last July. So when I read:

As for WDQS we only need to identify what version contains the security fix

If the search team make a new release with a security fix, and it is 32 releases ahead of the one currently pinned and regularly tested with, is it safe to apply that, even though it contains lots of non-security fixes? Or does it make sense to update the snapshots?

More generally (and possibly out of spec for this issue), is WDQS mainline going to be compatible with Wikibase 1.35 LTS for its lifespan? (Or however long WMDE supports it?) If not, will they support a branch? (Maybe there is already an understanding in this area, but I figure it's best to ask in case there isn't, since their "main customer" is on a rolling release and won't be impacted.)

As for announcements, the user group mailing list is quieter than the Telegram group; so yes, it's probably a good idea to use the list. Doing so might increase the number subscribed to it. But if the activity level there changes, it may start to make sense having a dedicated, announce-only list like MW.

Thanks for your thorough look at our work @GreenReaper !

If the search team make a new release with a security fix, and it is 32 releases ahead of the one currently pinned and regularly tested with, is it safe to apply that, even though it contains lots of non-security fixes? Or does it make sense to update the snapshots?

That's a good. Note that this task, and the linked document @toan drafted - meant for WMDE internal use, so might be not making all things explicit, deals with the security fixes specifically. That's what @toan didn't talk about "normal" changes to non-WMDE software like the query service.
This other topic we ponder under T272538, and there another internal doc draft linked from there that considers, among other things, the topic of non-security updates in non-WMDE software.
To put it (relatively) short: WMDE does not commit to very frequently update those components (i.e. outside of the general six-month release cycle). we might occasionally do it (e.g. when there is a significant performance improvement in Query Service or Elasticsearch that all Wikibase users would greatly benefit from - it might make sense to deliver those to users earlier than at the point of the next scheduled release. Given the limited resources WMDE has we cannot publish new versions of Wikibase suite whenever one of system changes. That also might not be intended by the users themselves - They probably wouldn't like to update their systems once or twice a week because some minor change happened in some deep backend component.

More generally (and possibly out of spec for this issue), is WDQS mainline going to be compatible with Wikibase 1.35 LTS for its lifespan? (Or however long WMDE supports it?)

For the time being WMDE will not be offering long-term support _Wikibase_ releases. We do not close a possibility of having Wikibase LTS version ("synced" with the Mediawiki LTS version) but we will decide on this once we have published a couple of "regular" releases and have seen the adoption of these and heard from the users on their needs.

If not, will they support a branch?

I am afraid I don't understand what "branch" you refer to? Do you mean Mediawiki's REL1_xx branches? If so, those Mediawiki (and other extensions) branches are indeed used in the release packages that WMDE will be publishing.

As for announcements, the user group mailing list is quieter than the Telegram group; so yes, it's probably a good idea to use the list. Doing so might increase the number subscribed to it. But if the activity level there changes, it may start to make sense having a dedicated, announce-only list like MW.

Good points. We do not want to overthink, so we will stick to the expertise of WMDE's Community Communication staff and use the communication channel(s) they recommend initially. Whatever we start with is of course a subject to normal evolution and will be changed and adopted according to the needs of Wikibase users.

Thanks for the extensive replies! That helps me understand your intent better - let me clarify my own:

I am afraid I don't understand what "branch" you refer to?

What I was getting at was a) the search team is outside of WMDE(?), so b) will they provide a branch of WDQS for security updates compatible with MW 1.35/6, rather than mainline, if necessary [i.e. if WMDE releases using 1.35/6]? Or will they agree to retain compatibility in their releases for what you're using, to let you quickly release a new package if they have a security-led WDQS update?

I don't yet fully understand how closely Wikibase is coupled with WDQS (I think "not much; mostly around updates"), but it strikes me that there is probably some interface point - and potentially other software included in your package may rely on specific behaviour that could have changed since the version of WDQS you packaged previously.

b) is also relevant to anyone, like myself, thinking of trying to run the Wikibase _extension_ and WDQS etc. separately. Specifically, to the question of whether it can be securely maintained on LTS, for fewer potentially disruptive upgrades when you e.g. have lots of other extensions - I gather there's been changes to services etc. in 1.36 - which in turn feeds into whether an installation can feasibly use MediaWiki LTS packaged by Debian; or extensions/skins not yet on 1.36.

It was my understanding that there's some intent to support 1.35, and there are commits along those lines, so that then led me to think "how will WDQS 1.35/6 work if WMF is focused on WDQS Wikidata on 1.37/wmf.x?"

But 1.35 might be out of scope for this issue if packages to be released by WMDE aren't intended to track MediaWiki's LTS branches in general. (Which I'd understand; it's a big piece of software, features are still in active development, and there are limited staff for LTS backports - which might anyway not be practical if they rely on new functionality.)

Hopefully that clears up what I was thinking, and also where I'm coming from: looking to add Wikibase to an existing MW setup; interested in your packages of components like WDQS (and hence, how they will be updated security-wise); but also in how the focus on packages affects development and security status of the Wikibase extension branches - because we may use it directly, and want to pick between MW 1.35 vs. 1.36 for an upcoming implementation.

Thanks for clarifications @GreenReaper !

I think it would make sense to differentiate between Wikibase (plus possible other Mediawiki extensions that WMDE "packs" together in the Wikibase Suite), and other Wikibase related software which is not a Mediawiki extension (eg. query service).
For Wikibase as a Mediawiki extension our thinking is the following: Each release Wikibase version is going to be compatible (i.e. WMDE will be ensuring this) with the last released Mediawiki version. So the Wikibase cersion we are going to release very soon will be compatible with MW 1.35. The next Wikibase release in Fall 2021 will be compatible with MW 1.36. And so on.
Possible Wikibase LTS version would be then "tracking" MW LTS version, but as said, for now it is just a consideration and WMDE does not have a clear plan about this.

Regarding things which are not Mediawiki extensions: I think you brought an important point about the query service. I think that it is justified to say that WDQS is really only compatible with the rolling version of Mediawiki/Wikibase which runs on WMF wikis (i.e. relevant wmf.x version). So the concern about possibly WDQS incompatiblity with "older" Wikibase versions is valid. WMDE is checking this by additional testing against the Wikibase version we intend to release WDQS with.
Luckily WDQS and Wikibase/Mediawiki are not very closely coupled so the risk of breaking the compatibility is not as big as in case of Mediawiki extension.

We will know better how that works in the context of the new WDQS updater in Fall, I guess.

I hope the above gives some answers to your questions.

Thanks, @WMDE-leszek. I see that plan outlined in T272478#7036933 and a proposed ADR. If I read it right, planned features that we're interested in trying such as mixed-ontology federation may only be usable on the stable release - which is soon to be MW 1.36. Along with WDQS, desire to reuse Lua from the ecosystem and the way we're rebuilding the site anyway, I'm leaning towards that.

For others, the decision might be different, and I'm glad to see the first new-style release made it out on MW 1.35, especially as it led to a few changes which might not otherwise have been backported. As you say, we'll see how it goes with WDQS - and once MW 1.39 arrives, demand for an LTS-oriented release may be clearer (perhaps others could provide it for clients with public patches, like Drupal).