Confirming approval from @JBennett on this plan moving forward

For context, the API will be used by the GrowthExperiments extension (see the ServiceImageRecommendationProvider class), on testwiki, arwiki, bnwiki, cswiki and viwiki. (See T290949: Add an image: Enable on test wikis for testwiki; others to come in a few weeks.) It's already used on the Beta Cluster equivalents beta enwiki. API requests are user-initiated; the data is displayed to the user after some transformations, it does not get stored (other than on the client-side for a short time). API outages would result in the relevant feature (one of the task types on Special:Homepage) showing an error dialog; somewhat annoying for evaluating A/B tests but not serious in terms of user impact. As such, there is no expected SLO.

Emergency shutdown is to set $wgGENewcomerTasksImageRecommendationsEnabled to false. We'll probably have an on-wiki shutdown mechanism as well by the time it goes to real wikis.

Ping @sdkim @JBennett for any updates as to various approvals and/or risk assessments.

@sbassett we are still using this POC and still plan to deprecate it ideally by March 31st (although we're not sure if we'll hit that target). Should this stay open?

In T294362#7716077, @sbassett wrote:

In T294362#7715894, @CBogen wrote:

@sbassett we are still using this POC and still plan to deprecate it ideally by March 31st (although we're not sure if we'll hit that target). Should this stay open?

Oh, sorry, I didn't realize this was an internal task for you. I've re-opened it. If you need Security-Team's input on anything, please feel free to re-tag the team.

@sbassett Thanks! Our understanding is that the security team wanted the old API decommissioned by March 31st because it's sitting on cloud services. Is this something you still care about?

In T294362#7716276, @CBogen wrote:

@sbassett Thanks! Our understanding is that the security team wanted the old API decommissioned by March 31st because it's sitting on cloud services. Is this something you still care about?

I don't think I was privy to any of those conversations, but as a general rule of thumb, I'd certainly encourage the full decommissioning of any tools or services which are no longer used or planned to no longer be used within the near future.

In T294362#7719390, @sbassett wrote:

In T294362#7716276, @CBogen wrote:

@sbassett Thanks! Our understanding is that the security team wanted the old API decommissioned by March 31st because it's sitting on cloud services. Is this something you still care about?

I don't think I was privy to any of those conversations, but as a general rule of thumb, I'd certainly encourage the full decommissioning of any tools or services which are no longer used or planned to no longer be used within the near future.

We are definitely planning to decommission the service once it's no longer in use, but we are still working on the infrastructure that will replace it, and extending the March 31st deadline would be helpful to us. Who on your team might know more about this?

In T294362#7719413, @CBogen wrote:

We are definitely planning to decommission the service once it's no longer in use, but we are still working on the infrastructure that will replace it, and extending the March 31st deadline would be helpful to us. Who on your team might know more about this?

Given the history of the task, I'd guess @JBennett? I'd say feel free to contact them directly or feel free to contact the more general [email protected].

Hey @CBogen -

Some AppSec folks (@Mstyles, @Reedy and myself) met today and have decided to grant the extension of the Image Suggestions POC for June 30th, 2022 under the following terms:

1. This will be entered into our risk register as a medium risk owned by you until the Image Suggestions POC is officially retired.
2. @JBennett has the prerogative to further review the deadline extension and increase or reduce the risk as they see fit.
3. We are assuming the Image Suggestions POC will not be enabled across any additional Wikimedia projects (as defined within wmgUseGrowthExperiments within IS.php) from now until its retirement.
4. As a best practice, this comment within CS.php should be updated to reflect this new deadline extension, in addition to any other code which assumes the previous retirement deadline of March 31st, 2022.

In T294362#7768458, @sbassett wrote:

Hey @CBogen -

Some AppSec folks (@Mstyles, @Reedy and myself) met today and have decided to grant the extension of the Image Suggestions POC to June 30th, 2022 under the following terms:

1. This will be entered into our risk register as a medium risk owned by you until the Image Suggestions POC is officially retired.
2. @JBennett has the prerogative to further review the deadline extension and increase or reduce the risk as they see fit.
3. We are assuming the Image Suggestions POC will not be enabled across any additional Wikimedia projects (as defined within wmgUseGrowthExperiments within IS.php) from now until its retirement.
4. As a best practice, this comment within CS.php should be updated to reflect this new deadline extension, in addition to any other code which assumes the previous retirement deadline of March 31st, 2022.

Great, thanks so much! These terms are fair and I'll work with the teams to update the code which assumes the previous retirement deadline.

Change 769779 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] CommonSettings: Update comment about Image Suggestions API

https://gerrit.wikimedia.org/r/769779

Change 769779 merged by jenkins-bot:

[operations/mediawiki-config@master] CommonSettings: Update comment about Image Suggestions API

https://gerrit.wikimedia.org/r/769779

Mentioned in SAL (#wikimedia-operations) [2022-03-10T21:39:15Z] <rzl@deploy1002> Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:769779|CommonSettings: Update comment about Image Suggestions API (T294362)]] (duration: 00m 48s)

In T294362#7768507, @CBogen wrote:

In T294362#7768458, @sbassett wrote:

Hey @CBogen -

Some AppSec folks (@Mstyles, @Reedy and myself) met today and have decided to grant the extension of the Image Suggestions POC to June 30th, 2022 under the following terms:

1. This will be entered into our risk register as a medium risk owned by you until the Image Suggestions POC is officially retired.
2. @JBennett has the prerogative to further review the deadline extension and increase or reduce the risk as they see fit.
3. We are assuming the Image Suggestions POC will not be enabled across any additional Wikimedia projects (as defined within wmgUseGrowthExperiments within IS.php) from now until its retirement.
4. As a best practice, this comment within CS.php should be updated to reflect this new deadline extension, in addition to any other code which assumes the previous retirement deadline of March 31st, 2022.

Great, thanks so much! These terms are fair and I'll work with the teams to update the code which assumes the previous retirement deadline.

We've merged patches in T306032: Adapt GrowthExperiments to new Image Suggestions API that will ride next week's train, arriving in group2 on Thursday June 30, and we've prepared config patches to start using the new API on that date.

So, in theory, this can all come together on June 30, in time to meet the deadline.

In practice, it's possible we'll need a few extra days to deal with train issues, or any errors that occur during the config switch – since there is no public facing API (T306349), it's hard to test this integration out thoroughly in advance.

@kostajh - Sounds good, thanks for the update. A few extra days beyond the deadline is fine. We just don't want that turning into weeks or months, so if the work appears to be heading in that direction due to unknowns, etc, please let us know so that we can recalibrate on the grant. Also - the AppSec team is still planning to complete T304885 by the end of this current quarter, or thereabouts.

In T294362#8026030, @sbassett wrote:

@kostajh - Sounds good, thanks for the update. A few extra days beyond the deadline is fine. We just don't want that turning into weeks or months, so if the work appears to be heading in that direction due to unknowns, etc, please let us know so that we can recalibrate on the grant. Also - the AppSec team is still planning to complete T304885 by the end of this current quarter, or thereabouts.

@sbassett we are aligned on not wanting this to drag on. That said, I think deploying the config patch to switch to the API would be better to do on Monday, so we have more time to deal with any issues that come up. So I'll plan on that, if that sounds OK to you.

In T294362#8030286, @kostajh wrote:

@sbassett we are aligned on not wanting this to drag on. That said, I think deploying the config patch to switch to the API would be better to do on Monday, so we have more time to deal with any issues that come up. So I'll plan on that, if that sounds OK to you.

Sounds fine, and gives us a little more time with the review anyways. Thanks.

Currently blocked on T312000: Unable to locate image suggestions for enwiki, and many hasrecommendation:image pages already have images, FYI. I'd like to see that resolved before we switch over to the new API.

In T294362#8048549, @kostajh wrote:

Currently blocked on T312000: Unable to locate image suggestions for enwiki, and many hasrecommendation:image pages already have images, FYI. I'd like to see that resolved before we switch over to the new API.

@sbassett haven't forgotten about this, just updating you that we're blocked on T312225: Envoy cannot connect to image-suggestion service, which will hopefully be resolved soon.

In T294362#8057536, @kostajh wrote:

In T294362#8048549, @kostajh wrote:

Currently blocked on T312000: Unable to locate image suggestions for enwiki, and many hasrecommendation:image pages already have images, FYI. I'd like to see that resolved before we switch over to the new API.

@sbassett haven't forgotten about this, just updating you that we're blocked on T312225: Envoy cannot connect to image-suggestion service, which will hopefully be resolved soon.

@sbassett we switched production traffic to the new API on July 7. Everything seems to be going fine so far. 🤞

We still use the old API on WCMS from beta cluster, hopefully that is OK to continue until T306349: Public-facing API for querying image suggestion recommendations and submitting user feedback is complete?

In T294362#8065058, @kostajh wrote:

@sbassett we switched production traffic to the new API on July 7. Everything seems to be going fine so far. 🤞

We still use the old API on WCMS from beta cluster, hopefully that is OK to continue until T306349: Public-facing API for querying image suggestion recommendations and submitting user feedback is complete?

Sounds great, thanks! As our risk register entry was only for temporarily using a wmcs-hosted resource within Wikimedia production, I will go ahead and "delete" the item from the registry, as it is no longer an issue. Using the API for beta cluster should be fine since it's wmcs (sorta) talking to wmcs.

Thanks @sbassett and @kostajh! Since we're going to continue to use the POC service on WCMS from beta cluster, should we close this ticket? cc @lbowmaker