Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Discrepancy with outcomes and return exit codes with failing shards #2496

Open
AaronMT opened this issue May 6, 2024 · 0 comments
Open

Discrepancy with outcomes and return exit codes with failing shards #2496

AaronMT opened this issue May 6, 2024 · 0 comments
Labels

Comments

@AaronMT
Copy link

AaronMT commented May 6, 2024

On the event of crashing shards (e.g, native Android crashes) reported, these are treated as device failures and not test result failures. Within matrix_ids.json, could report something like:

"axes": [
      {
        "device": "Pixel2.arm-30-en_US-portrait",
        "outcome": "failure",
        "details": "0 test cases failed, 595 passed, 2 flaky",
        "suiteOverview": {
          "total": 597,
          "errors": 0,
          "failures": 0,
          "flakes": 2,
          "skipped": 0,
          "elapsedTime": 264.07,
          "overheadTime": 0.0
        }
      }
    ]

Test result processing of matrix_ids.json and would be confused on this outcome with a failure reported with no failing tests.

FTL returns exit code of 10 with these failing shards (https://firebase.google.com/docs/test-lab/ios/command-line) wondering if an additional device descriptor can be appended to this outcome (if possible through FTL endpoints to have that data added to the axes)

@AaronMT AaronMT added the Bug label May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant