[Fleet] Fix failing/flaky tests #189559

kpollich · 2024-07-30T18:37:36Z

Summary

Closes #184310
Closes #180071
Closes #109017

Fixes several flaky/failing tests that were qualified as 8.15.0 blockers.

…ger incurs a mapping error

elasticmachine · 2024-07-30T18:37:39Z

Pinging @elastic/fleet (Team:Fleet)

obltmachine · 2024-07-30T18:37:52Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

kpollich · 2024-07-30T18:38:06Z

x-pack/test/fleet_api_integration/apis/epm/final_pipeline.ts

-            original: {
-              foo: 'bar',
-            },
+            original: JSON.stringify({ foo: 'bar' }),


The ES client seems to no longer coerce this value to a string under the hood, and event.original is mapped as keyword, so it needs to be a string.

kpollich · 2024-07-30T18:39:02Z

x-pack/test/fleet_api_integration/apis/epm/install_hidden_datastreams.ts

    skipIfNoDockerRegistry(providerContext);
    setupFleetAndAgents(providerContext);

    afterEach(async () => {
      await deletePackage('apm', '8.8.0');
    });

-    it('should rollover hidden datastreams when failed to update mappings', async function () {


The update between the two fixture versions of APM we maintain in the FTR directory no longer triggers a mapping conflict, likely because of upstream changes in Elasticsearch. Rather than try to find a new means to trigger a mapping conflict here, I think removing this in favor of the existing test coverage in Fleet's unit tests is sufficient.

kpollich · 2024-07-30T18:39:17Z

x-pack/test/fleet_functional/apps/home/welcome.ts

@@ -14,8  14,7 @@ export default function ({ getService, getPageObjects }: FtrProviderContext) {
  const PageObjects = getPageObjects(['common', 'home']);
  const kibanaServer = getService('kibanaServer');

-  // FLAKY: https://github.com/elastic/kibana/issues/109017
-  describe.skip('Welcome interstitial', () => {
+  describe('Welcome interstitial', () => {


Going to kick off a flaky test runner job for this one. It passed locally.

Flaky test runner job with 100 runs here: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6644

kibanamachine · 2024-07-30T19:44:22Z

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#6644

[❌] x-pack/test/fleet_functional/config.ts: 89/100 tests passed.

see run history

kpollich · 2024-07-30T19:49:22Z

Another flaky test runner here. From the screenshots in the failed runs it looked like the welcome test hadn't fully faded in by the time the timeout was reached.

https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6645

kibanamachine · 2024-07-30T20:56:01Z

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#6645

[✅] x-pack/test/fleet_functional/config.ts: 100/100 tests passed.

see run history

dmlemeshko · 2024-07-31T07:34:49Z

test/functional/page_objects/home_page.ts

+    await new Promise((resolve) => setTimeout(resolve, 1000));
+    return await this.testSubjects.isDisplayed('homeWelcomeInterstitial', 5000);


Not against the change, but if it is a page loading issue, you might need to update test with wait for global loading indicator to disappear:

await PageObjects.common.navigateToUrl('home', undefined, { disableWelcomePrompt: false }); await PageObjects.header.waitUntilLoadingHasFinished(); expect(await PageObjects.home.isWelcomeInterstitialDisplayed()).to.be(true);

Sometimes it may take a while to finish loading the page.

Thanks for taking a look @dmlemeshko - I've reverted the timeout bump and pulled in the header loading check instead. Rerunning flaky test runner here: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6652

criamico

LGTM

kibanamachine · 2024-07-31T15:47:52Z

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#6652

[✅] x-pack/test/fleet_functional/config.ts: 100/100 tests passed.

see run history

kpollich · 2024-07-31T16:24:25Z

@elasticmachine merge upstream

kibana-ci · 2024-07-31T17:52:21Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 305cd78

Failed CI Steps

FTR Configs #30

Metrics [docs]

✅ unchanged

History

💔 Build #225070 failed 6c5df45
💛 Build #224834 was flaky 14a5e5b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @kpollich

## Summary Closes elastic#184310 Closes elastic#180071 Closes elastic#109017 Fixes several flaky/failing tests that were qualified as 8.15.0 blockers. --------- Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit 4b701de)

kibanamachine · 2024-07-31T18:19:15Z

💚 All backports created successfully

Status	Branch	Result
✅	8.15

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

# Backport This will backport the following commits from `main` to `8.15`: - [[Fleet] Fix failing/flaky tests (#189559)](#189559)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Kyle Pollich <[email protected]>

kpollich added 3 commits July 30, 2024 14:08

Delete mapping conflict test as updating between test packages no lon…

43be6fd

…ger incurs a mapping error

Fix event.original tests

830ed52

Unskip welcome test

a473830

kpollich added release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v8.15.0 labels Jul 30, 2024

kpollich self-assigned this Jul 30, 2024

kpollich requested a review from a team as a code owner July 30, 2024 18:37

kpollich commented Jul 30, 2024

View reviewed changes

Bump timeouts for welcome test

14a5e5b

kpollich requested a review from a team as a code owner July 30, 2024 19:48

dmlemeshko reviewed Jul 31, 2024

View reviewed changes

criamico approved these changes Jul 31, 2024

View reviewed changes

Use header loading page object revert timeout bumps

6c5df45

Merge branch 'main' into fleet/flaky-tests-20240730

305cd78

kpollich merged commit 4b701de into elastic:main Jul 31, 2024
24 checks passed

kpollich deleted the fleet/flaky-tests-20240730 branch July 31, 2024 18:14

kibanamachine added the v8.16.0 label Jul 31, 2024

kibanamachine mentioned this pull request Jul 31, 2024

[8.15] [Fleet] Fix failing/flaky tests (#189559) #189667

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Fix failing/flaky tests #189559

[Fleet] Fix failing/flaky tests #189559

kpollich commented Jul 30, 2024 •

edited by kibanamachine

Loading

elasticmachine commented Jul 30, 2024

obltmachine commented Jul 30, 2024

kpollich Jul 30, 2024

kpollich Jul 30, 2024

kpollich Jul 30, 2024

kpollich Jul 30, 2024

kibanamachine commented Jul 30, 2024

kpollich commented Jul 30, 2024

kibanamachine commented Jul 30, 2024

dmlemeshko Jul 31, 2024

kpollich Jul 31, 2024

criamico left a comment

kibanamachine commented Jul 31, 2024

kpollich commented Jul 31, 2024

kibana-ci commented Jul 31, 2024

kibanamachine commented Jul 31, 2024

		await new Promise((resolve) => setTimeout(resolve, 1000));
		return await this.testSubjects.isDisplayed('homeWelcomeInterstitial', 5000);

[Fleet] Fix failing/flaky tests #189559

[Fleet] Fix failing/flaky tests #189559

Conversation

kpollich commented Jul 30, 2024 • edited by kibanamachine Loading

Summary

elasticmachine commented Jul 30, 2024

obltmachine commented Jul 30, 2024

🤖 GitHub comments

kpollich Jul 30, 2024

Choose a reason for hiding this comment

kpollich Jul 30, 2024

Choose a reason for hiding this comment

kpollich Jul 30, 2024

Choose a reason for hiding this comment

kpollich Jul 30, 2024

Choose a reason for hiding this comment

kibanamachine commented Jul 30, 2024

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#6644

kpollich commented Jul 30, 2024

kibanamachine commented Jul 30, 2024

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#6645

dmlemeshko Jul 31, 2024

Choose a reason for hiding this comment

kpollich Jul 31, 2024

Choose a reason for hiding this comment

criamico left a comment

Choose a reason for hiding this comment

kibanamachine commented Jul 31, 2024

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#6652

kpollich commented Jul 31, 2024

kibana-ci commented Jul 31, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

History

kibanamachine commented Jul 31, 2024

💚 All backports created successfully

Questions ?

kpollich commented Jul 30, 2024 •

edited by kibanamachine

Loading