Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inconsistent results from mergeFiles depending on number of cores #17

Open
zachary-foster opened this issue May 5, 2023 · 3 comments
Open

Inconsistent results from mergeFiles depending on number of cores #17

zachary-foster opened this issue May 5, 2023 · 3 comments

Comments

@zachary-foster
Copy link
Contributor

zachary-foster commented May 5, 2023 •
edited
Loading

Using the example command:

krisp_fasta test_data/krisp_fasta/ingroup*.gz --outgroup test_data/krisp_fasta/outgroup*.gz --conserved-left 30 --conserved-right 30 --diagnostic 40 --verbose --dot-alignment --cores 1 --w testtemp

I get different results each time it is run, either the expected 2 alignments, or no alignments. It seems random. This is related to the mergeFiles step and only happens on some number of processors:

  • 1, 2, 4: random result changes
  • 3, 5: works as expected
  • >5: hangs and never finishes

The version on krisp on pipy also has this problem, except that it does not hang with processors >5.
It may be related to the order that files are merged by mergeFiles.
@zachary-foster
Copy link
Contributor Author

Do you have any ideas on why this might occur @toopz?

zachary-foster added a commit that referenced this issue Jun 8, 2023
@zachary-foster
temporary fix to avoid inconsistent results from mergeKmerParallel
06271f7 
relates to #17
@zachary-foster
Copy link
Contributor Author

I looked more into this and may have narrowed down the issue some. I think the fault is with how mergeKmerParallel splits up input files for parallel execution. If I replace that function with a version that calls mergeKmerFiles without splitting up input files (not parallel), then it works as expected, except it still hangs with >5 cores (there are also 5 input files, hmmm), but I think this might be an unrelated issue.

@zachary-foster
Copy link
Contributor Author

The issue with hanging with >5 cores has been resolved (issue #20)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@zachary-foster