Adding TDD to false positives with test case to validate if model can… #124

zak-hassan · 2019-05-10T20:35:40Z

This PR is work in progress and shall be used to start a discussion on how we can implement this test case to validate our assumptions.

Description

This PR contains a unit test to validate if the model can learn from false positives. We are simulating the scenario of a human in the loop providing feedback that a log which is not an anomaly has be flagged incorrectly. Our model should learn from this and not report the same false prediction. We can filter programmatically but this is not machine learning. We should have the model learn from the data over time and get smarter.

What is included in this PR

Unit test to validate this.
What is missing is a function for passing in feedback into Self-organizing map model?

@MichaelClifford What do you recommend is the best approach in passing false positives into SOM?

ghost · 2019-05-10T20:38:02Z

Build failed.

thoth-coala : FAILURE in 38s
thoth-pytest : SUCCESS in 2m 00s

zak-hassan · 2019-05-10T20:45:29Z

The test case should be red and fail unless we are able to pass in the false positives into SOM and if its true that SOM can learn from false positive data.

ghost · 2019-05-10T20:45:31Z

Build failed.

thoth-coala : FAILURE in 39s
thoth-pytest : SUCCESS in 1m 49s

ghost · 2019-05-13T17:30:44Z

Build failed.

thoth-coala : FAILURE in 40s
thoth-pytest : FAILURE in 5m 29s

ghost · 2019-05-16T16:51:37Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.

ghost · 2019-05-16T17:07:16Z

Build failed.

thoth-coala : FAILURE in 37s
thoth-pytest : FAILURE in 2m 22s

ghost · 2019-05-16T18:46:39Z

Build failed.

thoth-coala : FAILURE in 44s
thoth-pytest : FAILURE in 2m 29s

ghost · 2019-05-17T14:21:43Z

Build failed.

thoth-coala : FAILURE in 39s
thoth-pytest : FAILURE in 2m 25s

ghost · 2019-05-17T14:40:46Z

Build failed.

thoth-coala : FAILURE in 42s
thoth-pytest : FAILURE in 2m 17s

ghost · 2019-05-17T14:56:08Z

Build failed.

thoth-coala : FAILURE in 35s
thoth-pytest : FAILURE in 6m 38s

zak-hassan · 2019-05-17T15:11:03Z

recheck

ghost · 2019-05-17T15:18:27Z

Build failed.

thoth-coala : FAILURE in 35s
thoth-pytest : FAILURE in 7m 02s

fridex · 2019-05-17T15:29:22Z

recheck

zak-hassan · 2019-05-17T15:34:12Z

@goern says "seems to be a problem between zuul and Openshift"

ghost · 2019-05-17T15:36:30Z

Build failed.

thoth-coala : FAILURE in 37s
thoth-pytest : FAILURE in 6m 50s

ghost · 2019-05-17T15:56:35Z

Build failed.

thoth-coala : FAILURE in 34s
thoth-pytest : FAILURE in 7m 14s

ghost · 2019-05-17T16:12:26Z

Build failed.

thoth-coala : FAILURE in 35s
thoth-pytest : FAILURE in 6m 43s

ghost · 2019-05-17T16:50:52Z

Build failed.

thoth-coala : FAILURE in 35s
thoth-pytest : FAILURE in 6m 58s

ghost · 2019-05-17T17:55:35Z

Build failed.

thoth-coala : FAILURE in 35s
thoth-pytest : FAILURE in 6m 47s

goern · 2019-05-18T19:30:44Z

recheck

ghost · 2019-05-18T19:37:49Z

Build failed.

thoth-coala : FAILURE in 37s
thoth-pytest : FAILURE in 6m 48s

goern · 2019-05-18T19:55:43Z

recheck

goern · 2019-05-18T20:00:57Z

@zmhassan do you know how much memory your pytests require?

ghost · 2019-05-18T20:02:59Z

Build failed.

thoth-coala : FAILURE in 34s
thoth-pytest : FAILURE in 6m 45s

goern · 2019-05-18T20:06:57Z

recheck

ghost · 2019-05-20T06:38:52Z

Build failed.

thoth-coala : FAILURE in 36s
thoth-pytest : FAILURE in 7m 16s

goern · 2019-05-20T10:38:59Z

recheck

ghost · 2019-05-20T10:40:07Z

Build failed.

thoth-coala : FAILURE in 42s
thoth-pytest : FAILURE in 20s

goern · 2019-05-20T10:48:53Z

recheck

ghost · 2019-05-20T10:56:53Z

Build failed.

thoth-coala : FAILURE in 38s
thoth-pytest : FAILURE in 6m 56s

goern · 2019-05-20T11:25:40Z

recheck

ghost · 2019-05-20T11:32:59Z

Build failed.

thoth-coala : FAILURE in 37s
thoth-pytest : FAILURE in 6m 59s

goern · 2019-05-21T09:49:50Z

recheck

ghost · 2019-05-21T09:57:24Z

Build failed.

thoth-coala : FAILURE in 39s
thoth-pytest : FAILURE in 7m 03s

ghost · 2019-05-21T17:28:32Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.

ghost · 2019-05-21T17:57:40Z

Build failed.

thoth-coala : FAILURE in 38s
thoth-pytest : FAILURE in 5m 23s

ghost · 2019-05-21T18:14:04Z

Build failed.

thoth-coala : FAILURE in 39s
thoth-pytest : FAILURE in 5m 15s

ghost · 2019-05-21T18:25:15Z

Build failed.

thoth-coala : FAILURE in 38s
thoth-pytest : FAILURE in 6m 36s

ghost · 2019-05-21T18:50:33Z

Build failed.

thoth-coala : FAILURE in 40s
thoth-pytest : FAILURE in 6m 56s

ghost · 2019-05-21T19:18:26Z

Build failed.

thoth-coala : FAILURE in 37s
thoth-pytest : FAILURE in 6m 48s

ghost · 2019-05-21T20:39:54Z

Build failed.

thoth-coala : FAILURE in 40s
thoth-pytest : FAILURE in 6m 55s

ghost · 2019-05-21T20:53:07Z

Build failed.

thoth-coala : FAILURE in 39s
thoth-pytest : FAILURE in 6m 31s

goern · 2019-05-22T06:23:40Z

Related Ticket: https://url.corp.redhat.com/b7d1e80

The 'module failure' seems to be related to a timeout between Zuul executor and the pod running pytest. We can reproduce the behaviour via a PR thoth-station/srcops-testing#109 and via an ansible playbook. The root cause might be the version of Ansible used by Zuul. to be continued...

ghost · 2019-05-23T15:25:22Z

Build failed.

thoth-coala : FAILURE in 37s
thoth-pytest : FAILURE in 2m 10s

ghost · 2019-05-23T15:29:42Z

Build failed.

thoth-coala : FAILURE in 38s
thoth-pytest : SUCCESS in 2m 43s

ghost · 2019-05-23T16:03:06Z

Build succeeded.

thoth-coala : SUCCESS in 37s
thoth-pytest : SUCCESS in 2m 27s

… learn from false positives

ghost · 2019-05-23T16:28:20Z

Build succeeded.

thoth-coala : SUCCESS in 37s
thoth-pytest : SUCCESS in 2m 42s

zak-hassan · 2019-05-23T16:28:51Z

@durandom So I removed the WIP and rebased. Awaiting on the merge ??

durandom

Thanks @zmhassan, looks good

zak-hassan requested review from durandom and MichaelClifford May 10, 2019 20:38

zak-hassan marked this pull request as ready for review May 10, 2019 20:48

zak-hassan force-pushed the feedback_loop branch from f8edc0a to 745f46e Compare May 16, 2019 16:51

zak-hassan force-pushed the feedback_loop branch from 745f46e to 3d31608 Compare May 16, 2019 17:04

zak-hassan force-pushed the feedback_loop branch from 16749a2 to f8edc0a Compare May 21, 2019 17:28

zak-hassan force-pushed the feedback_loop branch from f8edc0a to 16749a2 Compare May 21, 2019 17:45

Adding TDD to false positives with test case to validate if model can…

5d9a7cd

… learn from false positives

zak-hassan force-pushed the feedback_loop branch from 7ae4fb1 to 5d9a7cd Compare May 23, 2019 16:25

zak-hassan changed the title ~~[WIP] Adding TDD to false positives with test case to validate if model can…~~ Adding TDD to false positives with test case to validate if model can… May 23, 2019

durandom approved these changes May 23, 2019

View reviewed changes

durandom merged commit dc5d5c5 into AICoE:master May 23, 2019

Adding TDD to false positives with test case to validate if model can… #124

Adding TDD to false positives with test case to validate if model can… #124

Conversation

zak-hassan commented May 10, 2019 • edited Loading

Description

What is included in this PR

ghost commented May 10, 2019

zak-hassan commented May 10, 2019

ghost commented May 10, 2019

ghost commented May 13, 2019

ghost commented May 16, 2019

ghost commented May 16, 2019

ghost commented May 16, 2019

ghost commented May 17, 2019

ghost commented May 17, 2019

ghost commented May 17, 2019

zak-hassan commented May 17, 2019

ghost commented May 17, 2019

fridex commented May 17, 2019

zak-hassan commented May 17, 2019

ghost commented May 17, 2019

ghost commented May 17, 2019

ghost commented May 17, 2019

ghost commented May 17, 2019

ghost commented May 17, 2019

goern commented May 18, 2019

ghost commented May 18, 2019

goern commented May 18, 2019

goern commented May 18, 2019

ghost commented May 18, 2019

goern commented May 18, 2019

ghost commented May 20, 2019

goern commented May 20, 2019

ghost commented May 20, 2019

goern commented May 20, 2019

ghost commented May 20, 2019

goern commented May 20, 2019

ghost commented May 20, 2019

goern commented May 21, 2019

ghost commented May 21, 2019

ghost commented May 21, 2019

ghost commented May 21, 2019

ghost commented May 21, 2019

ghost commented May 21, 2019

ghost commented May 21, 2019

ghost commented May 21, 2019

ghost commented May 21, 2019

ghost commented May 21, 2019

goern commented May 22, 2019 • edited Loading

ghost commented May 23, 2019

ghost commented May 23, 2019

ghost commented May 23, 2019

ghost commented May 23, 2019

zak-hassan commented May 23, 2019

durandom left a comment

Choose a reason for hiding this comment

zak-hassan commented May 10, 2019 •

edited

Loading

goern commented May 22, 2019 •

edited

Loading