Generating datasets and training models for the first 18 wikis in this round went well. When I reached the Japanese Wiki, I kept running into the error shown in the screenshot below. To keep archives happy, this issue was solved by installing both mecab-python3 and unidic-lite in a venv.

Training models for the 4th round of wikis has been completed successfully.

We have also worked on models verification using the backtesting results shown below:

CCing @MGerlach, in case he'd like to add comments on the backtesting evaluation.

@kevinbazira great.
Most of the languages look fine but there are some redflags:

• jawiki with extremely low recall (0.01) and very low precision at 0.32
• aswiki with much lower precision than other wikis (0.57)

I assume that some of our heuristics are failing when working with these scripts. I dont have experience with these languages, so it is hard to pin down exactly, but I would speculate it is related to the tokenization when we generate anchor-candidates from substrings of the raw text. We would have to dig deeper here to understand what is the problem and how it could be fixed.
In the short run, I would recommend to not proceed with these languages.

@kostajh, we completed training models for the fourth round of wikis (listed in the task description) and shared the models' evaluation above that suggested we exclude jawiki and aswiki for the time being. We are now ready to publish the datasets for the wikis that passed the model evaluation, should we proceed?

In T304548#7938030, @kevinbazira wrote:

@kostajh, we completed training models for the fourth round of wikis (listed in the task description) and shared the models' evaluation above that suggested we exclude jawiki and aswiki for the time being. We are now ready to publish the datasets for the wikis that passed the model evaluation, should we proceed?

yes, please do. Thanks!

cc @Trizek-WMF about jawiki and aswiki.

We can exclude these wikis and proceed with the ones that are ready.
But what are the next steps for wikis we skip?

Since @MGerlach mentioned that some heuristics are failing when working with the skipped wikis, I think the next steps would be to manually inspect the models with users who have experience with these languages or use google-translate as the link-recommendation algorithm is iteratively improved until the models pass the backtesting evaluation.

In T304548#7952008, @kevinbazira wrote:

Since @MGerlach mentioned that some heuristics are failing when working with the skipped wikis, I think the next steps would be to manually inspect the models with users who have experience with these languages or use google-translate as the link-recommendation algorithm is iteratively improved until the models pass the backtesting evaluation.

Some thoughts on potential starting points on how to manually inspect the model. In order to figure where the text processing is failing for these languages, we could start checking the following steps:

• generation of candidate anchors (code). this uses simple tokenization to split the text into substrings which then serve as potential anchor-words which can be linked. if we cant identify suitable words, e.g. due to the absence of whitespaces, we wont be able to generate good link recommendations.
• generation of link candidates (code). for each candidate anchor, we look up link candidates in the anchor dictionary. The anchor dictionary contains all the already existing links (anchor title of the linked page) and is created in this script. we should make sure the anchor dictionary is populated with a sufficient number of links, otherwise we wont be able to generate link candidates for any anchor.
• disambiguation of link candidates (code). once we have one or more link candidates for a candidate anchor, the model selects the most probable link from the candidates. we can inspect the probabilities assigned to each link candidate to understand where a potential error might come from.

I'm keeping the two wikis in my freezer.

@MGerlach, when could we start checking on the models?

In T304548#7953084, @Trizek-WMF wrote:

I'm keeping the two wikis in my freezer.

@MGerlach, when could we start checking on the models?

Anytime. The models should be available for inspection locally on one of the stat-machines. @kevinbazira ran the training pipeline for these models and could probably share where the files for the models are located.
Maybe it would be worth to open a separate task for this work to be discussed in more detail? I think this could involve substantial amount of effort to find the error and try to fix it since we dont yet know what exactly we are looking for.

I have opened a separate task here T309263 and shared the location of the models.

Mentioned in SAL (#wikimedia-operations) [2022-05-26T19:40:28Z] <tgr> T304548 running extensions/GrowthExperiments/maintenance/changeWikiConfig.php on tier4 Growth wikis

Populated section title config (and documented the process).
abwiki, adywiki, akwiki and atjwiki do not have other task types at all:

Change 800247 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[operations/mediawiki-config@master] Enable GrowthExperiments link recommendations, round 4

https://gerrit.wikimedia.org/r/800247

Change 800247 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable GrowthExperiments link recommendations, round 4

https://gerrit.wikimedia.org/r/800247

Mentioned in SAL (#wikimedia-operations) [2022-05-26T20:12:44Z] <brennen@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:800247|Enable GrowthExperiments link recommendations, round 4 (T304548)]] (duration: 00m 56s)

In T304548#7961621, @Tgr wrote:

Populated section title config (and documented the process).
abwiki, adywiki, akwiki and atjwiki do not have other task types at all:

@Trizek-WMF is that normal?

Yes. When we ran the deployment by default of Growth features to all Wikipedias, we found out that some of the smaller ones don't have any recognizable maintenance templates, or no maintenance templates at all. "Add a link" is the solution we have for these wikis.

Wikis with no suggested edits (cc @kostajh):

• ak
• as

The other ones have suggested edits.

In T304548#7991680, @Trizek-WMF wrote:

Wikis with no suggested edits (cc @kostajh):

• ak
• as

The other ones have suggested edits.

I see suggested edits on as.wikipedia.org:

The only task type for akwiki is link-recommendation, which won't be visible until we enable the front-end.

@kostajh, I went to Special:Search, used hasrecommendation:link to find articles and had nothing there. Are we good?

In T304548#7998561, @Trizek-WMF wrote:

@kostajh, I went to Special:Search, used hasrecommendation:link to find articles and had nothing there. Are we good?

@Trizek-WMF which wiki(s) are you referring to?

@kostajh, I used the process listed in the task description:

1. In Special:Search, use hasrecommendation:link to find articles
2. Test them on https://api.wikimedia.org/service/linkrecommendation/apidocs/#/default/get_v1_linkrecommendations__project___domain___page_title_
3. In Special:NewcomerTasksInfo, link-recommendation gives the number of articles

When I checked, the deployment list (minus as and ja), the only two wikis without Links suggestions are ak and as.

In T304548#7999400, @Trizek-WMF wrote:

@kostajh, I used the process listed in the task description:

1. In Special:Search, use hasrecommendation:link to find articles
2. Test them on https://api.wikimedia.org/service/linkrecommendation/apidocs/#/default/get_v1_linkrecommendations__project___domain___page_title_
3. In Special:NewcomerTasksInfo, link-recommendation gives the number of articles

When I checked, the deployment list (minus as and ja), the only two wikis without Links suggestions are ak and as.

Thanks for that.

It looks like aswiki wasn't specified in the config patch https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/ /800247, so I suspect adding it there would fix the problem there

akwiki was in the deployment, so checking the logs ( grep akwiki /var/log/mediawiki/mediawiki_job_growthexperiments-refreshLinkRecommendations-s*/syslog.log), I see:

mediawiki_job_growthexperiments-refreshLinkRecommendations-s3/syslog.log:Jun 15 07:13:05 mwmaint1002 mediawiki_job_growthexperiments-refreshLinkRecommendations-s3[1057]: akwiki:      checking candidate Facebook... number of good links too small (1)
mediawiki_job_growthexperiments-refreshLinkRecommendations-s3/syslog.log:Jun 15 07:13:06 mwmaint1002 mediawiki_job_growthexperiments-refreshLinkRecommendations-s3[1057]: akwiki:      checking candidate YouTube... All of the links in the recommendation have been pruned
mediawiki_job_growthexperiments-refreshLinkRecommendations-s3/syslog.log:Jun 15 07:13:06 mwmaint1002 mediawiki_job_growthexperiments-refreshLinkRecommendations-s3[1057]: akwiki:      checking candidate Etuo... All of the links in the recommendation have been pruned
mediawiki_job_growthexperiments-refreshLinkRecommendations-s3/syslog.log:Jun 15 07:13:06 mwmaint1002 mediawiki_job_growthexperiments-refreshLinkRecommendations-s3[1057]: akwiki:      topic exhausted, 500 tasks still needed

Given that there are only 590 content pages, I suspect there is not enough content on this wiki to generate link recommendations. @Trizek-WMF should we remove it from the deployment list?

Thank you for checking, Kosta.

I moved akwiki to the next round, hopefully it won't be forgotten twice! ;)

Regarding wikis with a low number of articles, it is a problem. The majority of these wikis have no maintenance templates, hence no way to offer tasks to newcomers. When we deployed the Growth features there, our plan with Marshall was to have Add a link being deployed, and then to offer at least one task type to newcomers.

What should we do, @MMiller_WMF? give up on these wikis, or find a way to improve the model?

Change 805766 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] GrowthExperiments: Enable link recommendation on aswiki

https://gerrit.wikimedia.org/r/805766

@Trizek-WMF should we proceed with enabling the front-end for the wikis today?

In T304548#8005893, @Trizek-WMF wrote:

Thank you for checking, Kosta.

I moved akwiki to the next round, hopefully it won't be forgotten twice! ;)

To clarify: akwiki wasn't forgotten, it's just that our script does not generate any link recommendations on this wiki because there is not enough content (see T304548#8005057 for details).

Do you want to move aswiki to the next round? That one was indeed missed in this round.

In T304548#8008475, @kostajh wrote:

@Trizek-WMF should we proceed with enabling the front-end for the wikis today?

In T304548#8005893, @Trizek-WMF wrote:

Thank you for checking, Kosta.

I moved akwiki to the next round, hopefully it won't be forgotten twice! ;)

To clarify: akwiki wasn't forgotten, it's just that our script does not generate any link recommendations on this wiki because there is not enough content (see T304548#8005057 for details).

Do you want to move aswiki to the next round? That one was indeed missed in this round.

@Trizek-WMF @KStoller-WMF no deploys on Fridays, so the next opportunity is Monday. Please comment here if we should go ahead with enabling the frontend for add link for these wikis, with the exception of aswiki and akwiki which are in the next round.

Change 806365 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] GrowthExperiments: Enable link recommendations frontend, round 4

https://gerrit.wikimedia.org/r/806365

In T304548#8010798, @kostajh wrote:

@Trizek-WMF @KStoller-WMF no deploys on Fridays, so the next opportunity is Monday. Please comment here if we should go ahead with enabling the frontend for add link for these wikis, with the exception of aswiki and akwiki which are in the next round.

Let's go ahead on Monday.

If we can have as.wp in this round, then let's go. Otherwise, it is fine to have it with the next round.

In T304548#8011141, @Trizek-WMF wrote:

In T304548#8010798, @kostajh wrote:

@Trizek-WMF @KStoller-WMF no deploys on Fridays, so the next opportunity is Monday. Please comment here if we should go ahead with enabling the frontend for add link for these wikis, with the exception of aswiki and akwiki which are in the next round.

Let's go ahead on Monday.

Oops, there are no deploys on Monday due to the earth day holiday.

In T304548#8011145, @Trizek-WMF wrote:

If we can have as.wp in this round, then let's go. Otherwise, it is fine to have it with the next round.

What we could do is: enable the backend for aswiki on Tuesday, and enable the frontend for all 4th round wikis on Thursday. That way aswiki has ~2 days for the task pool to fill up. Sounds OK?

Sounds ok! :)

Change 805766 merged by jenkins-bot:

[operations/mediawiki-config@master] GrowthExperiments: Enable link recommendation on aswiki

https://gerrit.wikimedia.org/r/805766

Change 806365 merged by jenkins-bot:

[operations/mediawiki-config@master] GrowthExperiments: Enable link recommendations frontend, round 4

https://gerrit.wikimedia.org/r/806365

Mentioned in SAL (#wikimedia-operations) [2022-06-23T07:25:21Z] <samtar@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:806365|GrowthExperiments: Enable link recommendations frontend, round 4 (T304548)]] (duration: 03m 37s)

In T304548#8011193, @kostajh wrote:

In T304548#8011141, @Trizek-WMF wrote:

In T304548#8010798, @kostajh wrote:

@Trizek-WMF @KStoller-WMF no deploys on Fridays, so the next opportunity is Monday. Please comment here if we should go ahead with enabling the frontend for add link for these wikis, with the exception of aswiki and akwiki which are in the next round.

Let's go ahead on Monday.

Oops, there are no deploys on Monday due to the earth day holiday.

In T304548#8011145, @Trizek-WMF wrote:

If we can have as.wp in this round, then let's go. Otherwise, it is fine to have it with the next round.

What we could do is: enable the backend for aswiki on Tuesday, and enable the frontend for all 4th round wikis on Thursday. That way aswiki has ~2 days for the task pool to fill up. Sounds OK?

Sorry, I should have read farther back up the comment thread here.

• in T304548#7953084 we agreed to hold back aswiki and jawiki due to issues with the models
• We have T309263: Support languages whose add-a-link models were not published for investigating the model further

So, we've deployed the fourth round without those two wikis.

Thank you!

Waiting for QA to close this ticket.