Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flakes attribute never set correctly in FullJunitReport.xml #2504

Open
AaronMT opened this issue Jun 24, 2024 · 13 comments · May be fixed by #2536
Open

Flakes attribute never set correctly in FullJunitReport.xml #2504

AaronMT opened this issue Jun 24, 2024 · 13 comments · May be fixed by #2536
Labels
Bug

Comments

@AaronMT
Copy link

AaronMT commented Jun 24, 2024 •
edited
Loading 

<testsuite name="Pixel2.arm-30-en_US-portrait" tests="462" failures="1" flakes="0" errors="0" skipped="0" time="10277.484" timestamp="2024-06-23T06:02:53" hostname="localhost">
    <testcase name="verifyButtonTest" classname="org.app.verifyButtonTestTest" time="0.222" flaky="true">
     ...
</testsuite>

Shouldn't flakes attribute be 1 in this case? This is with v23.10.1.

Our Flank config specifies num-flaky-test-attempts: 1 with full-junit-result: true under the Flank section.
@AaronMT AaronMT added the Bug label Jun 24, 2024
@SelaseKay
Copy link
Collaborator

SelaseKay commented Oct 11, 2024 •
edited
Loading

Hi @AaronMT , thanks for the report. I'm able to reproduce this issue. I'm looking into it.

@SelaseKay
Copy link
Collaborator

Hi @AaronMT , I was able to reproduce this once but for some weird reason, I'm not able to reproduce it anymore. Are you still experiencing this issue?

@AaronMT
Copy link
Author

AaronMT commented Oct 16, 2024 •
edited
Loading

Yes, I still see this issue on 23.10.1. Here's an example report from our CI from yesterday.

<testcase name="verifyReaderModeControlsTest" classname="org.mozilla.fenix.ui.ReaderViewTest" time="0.275" flaky="true">
<failure>
java.security.ProviderException: Keystore operation failed at ...
</failure>
<webLink>
...
</webLink>
</testcase>

Where flaky is set to true, but the top level testsuite has the attributeflakes set to 0

<testsuite name="Pixel2.arm-30-en_US-portrait" tests="486" failures="0" flakes="0" errors="0" skipped="0" time="10298.759" timestamp="2024-10-14T21:30:46" hostname="localhost">

@SelaseKay
Copy link
Collaborator

Are you able to reproduce this locally?

@SelaseKay
Copy link
Collaborator

Can you provide a sample app and test apk?

@AaronMT
Copy link
Author

AaronMT commented Oct 17, 2024

As Flank (via Test Lab) generates these artifacts, I'm not sure what you mean by reproducing locally.

Here's a recent (arm64v8a debug build from our CI) and no-arch test APK

https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/M4IjtgpLTWuZaUg43Q0ByA/artifacts/public/build/target.arm64-v8a.apk

https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/TvdzLJF3Tiij13O8gqe9dA/artifacts/public/build/target.noarch.apk

@SelaseKay
Copy link
Collaborator

I apologise for the confusion. By reproducing locally, I meant executing Flank on your own machine to initiate Test Lab runs, rather than in CI.

@SelaseKay
Copy link
Collaborator

Kindly provide the device specifications you have in your flank.yml

@SelaseKay
Copy link
Collaborator

SelaseKay commented Oct 17, 2024 •
edited
Loading

Seems to work fine locally:

<?xml version='1.0' encoding='UTF-8' ?>
<testsuites>
  <testsuite name="Pixel2.arm-30-en_US-portrait" tests="45" failures="11" flakes="1" errors="0" skipped="0" time="575.163" timestamp="2024-10-17T20:08:37" hostname="localhost">
    <testcase name="testExperimentUnenrolledViaSecretMenu" classname="org.mozilla.fenix.experimentintegration.GenericExperimentIntegrationTest" time="50.552">
      <failure>androidx.test.espresso.NoActivityResumedException: No activities in stage RESUMED. Did you forget to launch the activity. (test.getActivity() or similar)?
	at dalvik.system.VMStack.getThreadStackTrace(Native Method)
	at java.lang.Thread.getStackTrace(Thread.java:1736)
      ......
    </testcase>
    <testcase name="settingsTest" classname="org.mozilla.fenix.screenshots.ComposeMenuScreenShotTest" time="10.278" flaky="true">
      <failure>FAILED
    </failure>
......
    </testcase>
  </testsuite>
</testsuites>

@AaronMT
Copy link
Author

AaronMT commented Oct 21, 2024 

gcloud:
  results-bucket: ...
  record-video: true
  timeout: 15m
  async: false
  num-flaky-test-attempts: 1

  app: /app/path
  test: /test/path

  auto-google-login: false
  use-orchestrator: true
  environment-variables:
    clearPackageData: true
  performance-metrics: true

  test-targets:
    - notPackage org.mozilla.fenix.screenshots
    - notPackage org.mozilla.fenix.syncintegration
    - notPackage org.mozilla.fenix.experimentintegration

  device:
    - model: Pixel2.arm
      version: 30
      locale: en_US

flank:
  project: ...
  max-test-shards: 100
  num-test-runs: 1
  output-style: compact
  full-junit-result: true

Does this have anything to do with full-junit-result?

@SelaseKay
Copy link
Collaborator

Do you experience this issue when you run flank in your local environment with the above configuration?

@AaronMT
Copy link
Author

AaronMT commented Oct 23, 2024 •
edited
Loading

Hello again, yes I was able to reproduce with a local Flank run call to the same configuration above.

I had a flaky test in my run below with the following top level test suite

<testsuite name="MediumPhone.arm-34-en_US-portrait" tests="498" failures="0" flakes="0" errors="0" skipped="0" time="11272.089" timestamp="2024-10-23T09:46:12" hostname="localhost">
<testcase name="verifyCFRAfterBlockingTheCookieBanner" classname="org.mozilla.fenix.ui.CookieBannerBlockerTest" time="38.066" flaky="true">
<failure>
java.lang.AssertionError: UiSelector[CONTAINS_TEXT=Less distractions, less cookies tracking you on this site.] does not exist at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.assertTrue(Assert.java:42) at org.mozilla.fenix.helpers.MatcherHelper.assertUIObjectExists(MatcherHelper.kt:100) at org.mozilla.fenix.ui.robots.BrowserRobot.verifyCookieBannerBlockerCFRExists(BrowserRobot.kt:833) at org.mozilla.fenix.ui.CookieBannerBlockerTest$verifyCFRAfterBlockingTheCookieBanner$1$4.invoke(CookieBannerBlockerTest.kt:60) at org.mozilla.fenix.ui.CookieBannerBlockerTest$verifyCFRAfterBlockingTheCookieBanner$1$4.invoke(CookieBannerBlockerTest.kt:58) at org.mozilla.fenix.ui.robots.NavigationToolbarRobot$Transition.enterURLAndEnterToBrowser(NavigationToolbarRobot.kt:226) at org.mozilla.fenix.ui.CookieBannerBlockerTest$verifyCFRAfterBlockingTheCookieBanner$1.invoke(CookieBannerBlockerTest.kt:58) at org.mozilla.fenix.ui.CookieBannerBlockerTest$verifyCFRAfterBlockingTheCookieBanner$1.invoke(CookieBannerBlockerTest.kt:45) at org.mozilla.fenix.helpers.AppAndSystemHelper.runWithCondition(AppAndSystemHelper.kt:686) at org.mozilla.fenix.ui.CookieBannerBlockerTest.verifyCFRAfterBlockingTheCookieBanner(CookieBannerBlockerTest.kt:45)
</failure>
<webLink>
https://console.firebase.google.com/project/moz-fenix/testlab/histories/bh.66b7091e15d53d45/matrices/7524624672444898348/executions/bs.a2c09c06d295e45c/testcases/2
</webLink>
</testcase>

@SelaseKay
Copy link
Collaborator

SelaseKay commented Oct 31, 2024 •
edited
Loading

Hi @AaronMT, after some investigation, I've found that the issue is related to the max-test-shards property. If you omit this property, allowing it to use it's default value, the behavior is as expected (with flakes reflecting the correct value).

@SelaseKay SelaseKay linked a pull request Oct 31, 2024 that will close this issue
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: resolve flaky tests counts discrepancy SelaseKay/flank
2 participants
@AaronMT @SelaseKay