Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A way to hook before/after job execution (and/or a way to see a number of tasks in the processing queue) #1170

Open
nagisa opened this issue May 29, 2024 · 0 comments

Comments

@nagisa
Copy link

nagisa commented May 29, 2024

An application I'm working on that uses rayon has gotten to a state where, if the CPU is constantly utilized to the max, the memory keeps growing in an unbounded manner. We have troubleshooted this down to the fact that we keep putting all sorts of auxiliary tasks to said queue and the computer is simply unable to keep up with the inflow of those tasks.

I would like to make sure this sort of an issue does not require another extended investigation. In order to achieve this, I would like to add a counter metric or some other exposition of the current rayon's queue depth.

At the most basic level a simple accessor function would be good enough, but I think it might be not be particularly convenient to replicate this information to, say, prometheus metrics.

A much more convenient option would be if the pools provided an ability to register some callbacks similar to the builder methods that allow tweaking how the threads are spawned.

For example in my case:

  1. A callback that's fired whenever a job is pushed into the queue;
  2. A callback when a job is popped and/or whenever pool finishes executing code for a job;
  3. One could imagine further additional such callbacks (e.g. whenever a job execution begins...)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant