-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parallelize the process #8
Comments
Another option discussed during the biohackathon would be to split the source dump in the same number of parts as CPUs, run the dockers in parallel in the different CPUs, and join the results. @nilshoffmann |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
At this moment the dump processor seems to be sequential and uses only one of the 60 cores in our server.
This can be seen with
htop
It would be nice to do some research about how we can parallelize the dump processing code. Some questions that I would like to answer:
The text was updated successfully, but these errors were encountered: