This'll require the following team-specific infrastructure:

Why is this team-specific? Surely every team with browser tests would eventually like them to be running on every patch.
Why is the wheel being re-invented? T101068 and T101069 are both already implemented by zuul and jenkins, T101063 is done by the CI infrastructure for qunit, and might need some adjustments for browser tests.

@Legoktm hopefully discussion on T101325 will help. From talking to Release engineering the plan is to have this for every project, but we want to pioneer and explore getting this now as this seems to be a year or so away in the roadmap.

Update: So if we get the labs instance and the Gerrit account, I estimate writing the script to take no more than an afternoon (all it should involve is something to watch changes and check them out and then run tests and then use gerrit.py for results - this part is mostly done in T101346)
Seems there is a little resistance on e-mail thread that we need to take into account before doing this.

Why is this team-specific? Surely every team with browser tests would eventually like them to be running on every patch.

I've removed that awkward/incorrect wording from the description. Ta.

In T100293#1335913, @Jdlrobson wrote:

@Legoktm hopefully discussion on T101325 will help. From talking to Release engineering the plan is to have this for every project, but we want to pioneer and explore getting this now as this seems to be a year or so away in the roadmap.

Didn't really help. Why is the wheel being re-invented? Lets start with a) why is zuul not sufficient for talking to gerrit?

I'm not sure what is being reinvented here. We run browser tests daily on beta labs on master. We want to run browser tests per commit on a non beta labs instance. As far as I understand there is no way to spin up an instance for a given commit and run browser tests. If that's not true that's great and please point me @ relevant docs.

Maybe we should chat over this in irc/hangout if there are some crossed wires.

In T100293#1354526, @Jdlrobson wrote:

I'm not sure what is being reinvented here. We run browser tests daily on beta labs on master. We want to run browser tests per commit on a non beta labs instance.

I commented on T101069#1327976, but was told to go here...the task says "When a patch is submitted to Gerrit trigger a script". This is literally what zuul jenkins do. When a patch is submitted to a specific repository, zuul in turn triggers a jenkins job, which runs whatever defined shell code the job definition says. So, why is that solution inadequate to the point where the wheel needs to be re-invented?

As far as I understand there is no way to spin up an instance for a given commit and run browser tests. If that's not true that's great and please point me @ relevant docs.

When you say instance, do you want a brand new labs instance for each commit? If so, why? Is just a new mediawiki installation sufficient?

Sorry! I've updated the title - it was maybe confusingly worded - yes zuul jenkins could be the solution to that - we've not investigated the solution to it yet. A new mediawiki instance should be fine - the browser tests only have a few dependencies needed - such as Echo extension and Thanks.

Sounds like we could use zuul in some way - I'm not too familiar with the how yet though so any help fleshing out how we can achieve the task in the main description would be much appreciated.

Please see [Update] Browser tests per patch -email.

@Jdlrobson Hey I remember you told me the bot was running now automatically picking up new patches on gerrit.

Is that true? What else are we missing? (I'm creating cards for the next sprint https://phabricator.wikimedia.org/tag/reading-web-sprint-51-x/). The main thing I can think of is expose the bot logs somewhere so that we can check what it's doing without having to log into the instance. What do you think?

I know that @dduvall has been experimenting around this and made some progress https://integration.wikimedia.org/ci/job/mwext-MobileFrontend-mw-selenium/42/consoleFull

It would be worth checking in with him, but the only thing outstanding in terms of my script if we want/need to continue down that road is:

• Make the bot real time and run on submitting a patch rather than potentially up to 30 minutes delay
• Post results somewhere and link to it from Gerrit review
• Make it easier to spin up an instance with Barry - e.g. vagrant role

In T100293#1422770, @Jdlrobson wrote:

I know that @dduvall has been experimenting around this and made some progress https://integration.wikimedia.org/ci/job/mwext-MobileFrontend-mw-selenium/42/consoleFull

Created T104840 for us to actually devote time to that this sprint.

It would be worth checking in with him, but the only thing outstanding in terms of my script if we want/need to continue down that road is:

• Make the bot real time and run on submitting a patch rather than potentially up to 30 minutes delay

I'm guessing this is T101069, so - tracked ☑️.

• Post results somewhere and link to it from Gerrit review

Created T104841 to track this.

• Make it easier to spin up an instance with Barry - e.g. vagrant role

Created T104842 for tracking automation. All this will also be part of how we resolve T104562 as part of making this easily reproducible.

In T100293#1429234, @Jhernandez wrote:

In T100293#1422770, @Jdlrobson wrote:

I know that @dduvall has been experimenting around this and made some progress https://integration.wikimedia.org/ci/job/mwext-MobileFrontend-mw-selenium/42/consoleFull

Created T104840 for us to actually devote time to that this sprint.

Just an update on the mwext-mw-selenium JJB job that it now supports video recording of the Xvfb session (woot): Video files are saved as Jenkins build artifacts for each failed scenario—removed after 3 days to save disk space.[1][2]

The job seems fairly stable (there was a small fix to the supporting slave scripts today, but the MVP in T103039 is now considered done) and is already scheduled under the experimental pipeline for MobileFrontend. Scheduling it for Gather or any other MW extension that uses MW-Selenium 1.4 would only a matter of one line in the JJB/Zuul config.

What other functionality are you looking to achieve with Barry, and can we maybe pool our efforts to bring it into the JJB builder? AFAICT, the triggering and reporting aspects detailed in T101069 are already covered by the latter given that it's run as a Zuul-scheduled Jenkins job, but I could be missing other aspects of the project.

[1] https://gerrit.wikimedia.org/r/#/c/227584/
[2] https://www.mediawiki.org/wiki/Continuous_integration/Browser_tests#Video_recordings_and_screenshots

Ping @Jdlrobson ^. What do we use barry for that is not done by JJB? At what point would it make sense to use JJB instead of barry?

The video recording sounds crazy awesome hah.

Yup let's explore it - T107587
Provided we can cover all the tests I think we are good to scrap Barry in existing form and use him for something else :).

Awesome!

Been pinging around to get resolution with the last two tasks blocking. It'd be great to get this one out of the way already.

Venimus vidimus vicimus!