Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement injecting sleep in go failpoints #14729

Open
Tracked by #14045
serathius opened this issue Nov 10, 2022 · 10 comments
Open
Tracked by #14045

Implement injecting sleep in go failpoints #14729

serathius opened this issue Nov 10, 2022 · 10 comments
Assignees
Labels
help wanted priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. stage/tracked type/feature

Comments

@serathius
Copy link
Member

What would you like to be added?

Linearization tests only support injecting panic https://github.com/etcd-io/etcd/blob/main/tests/linearizability/failpoints.go#L33-L53, it would be useful to extend it to also support injecting a sleep. It should already be supported by gofail.

Main change that needs to be implemented is detecting that failpoint was executed. For panic it's simple, we wait for process to exit. Injecting sleep is much more subtle. I think this can be achieved by listing the failpoints via http endpoint.

Why is this needed?

This will allow for easier detection of race conditions in etcd code.

@serathius serathius changed the title Allow injecting sleep in go failpoints Implement injecting sleep in go failpoints Nov 10, 2022
@serathius serathius added help wanted priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Nov 10, 2022
@ramil600
Copy link
Contributor

I started working on this, and then noticed that gofail doesn't disable the fail points properly through the http DELETE request:
https://github.com/etcd-io/gofail/issues/26
PR submitted and linked to the same issue.

@serathius
Copy link
Member Author

Thanks for looking into this!

@ramil600
Copy link
Contributor

ramil600 commented Nov 16, 2022

@serathius, sure no problem, currently I can register sleep fail point and check whether it was successfully enabled. However I am not sure when it should be deactivated. Once sleep is enabled in the code, it will trigger sleep every time when it is executed. PR: #14796

@serathius
Copy link
Member Author

Wrote on #14796 that we care about failpoint execution not failpoint enablement.

@ramil600
Copy link
Contributor

@serathius, one suggestion I have is resetting sleep duration value to 0 and deleting description in the terms field of the failpoint, once sleep is executed.
Then when you check it through http endpoint, it still exists(http status = 200), but it won't execute sleep anymore, unless randomly activated again. Please let me know if it is ok with you and I will send the PR to modify gofail library.

@lavacat
Copy link

lavacat commented Jan 4, 2023

@ramil600 Are you still working on this issue? I think this is going to be very useful.

@pchan
Copy link
Contributor

pchan commented May 8, 2023

I can take this up and can build on ramil600 PR : #14796

@lavacat lavacat assigned pchan and unassigned ramil600 May 8, 2023
@lavacat
Copy link

lavacat commented May 8, 2023

@pchan assigned to you. To fix this you'll need etcd-io/gofail#37 first

@lavacat
Copy link

lavacat commented May 8, 2023

@ramil600 apologies for reassigning without checking with you first.
I think you've made a good progress on etcd-io/gofail#37. Hope we can merge it soon and unblock this issue.

@stale
Copy link

stale bot commented Aug 12, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. stage/tracked type/feature
Development

No branches or pull requests

4 participants