I am also getting a similar input file name collision error when NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:GATK4_MARKDUPLICATES is called:

Aug-08 16:16:14.925 [Actor Thread 456] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:GATK4_MARKDUPLICATES (2); work-dir=null error [nextflow.exception.ProcessUnrecoverableException]: Process NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:GATK4_MARKDUPLICATES input file name collision -- There are multiple input files for each of the following file names: Tumor1-L3.0010.bam, Tumor1-L3.0001.bam, Tumor1-L3.0007.bam, Tumor1-L3.0008.bam, Tumor1-L3.0004.bam, Tumor1-L3.0012.bam, Tumor1-L3.0003.bam, Tumor1-L3.0009.bam, Tumor1-L3.0005.bam, Tumor1-L3.0011.bam, Tumor1-L3.0002.bam, Tumor1-L3.0006.bam

nextflow.log

I am also getting a similar input file name collision error when NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:GATK4_MARKDUPLICATES is called:

Aug-08 16:16:14.925 [Actor Thread 456] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:GATK4_MARKDUPLICATES (2); work-dir=null error [nextflow.exception.ProcessUnrecoverableException]: Process NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:GATK4_MARKDUPLICATES input file name collision -- There are multiple input files for each of the following file names: Tumor1-L3.0010.bam, Tumor1-L3.0001.bam, Tumor1-L3.0007.bam, Tumor1-L3.0008.bam, Tumor1-L3.0004.bam, Tumor1-L3.0012.bam, Tumor1-L3.0003.bam, Tumor1-L3.0009.bam, Tumor1-L3.0005.bam, Tumor1-L3.0011.bam, Tumor1-L3.0002.bam, Tumor1-L3.0006.bam

nextflow.log

@brandon-hastings : Which version of Sarek are you using? Can you reproduce the error with the latest v3.4.3?

I was using v3.4.2, but I ran the pipeline again with v3.4.3 and received the same error.

I was able to look into this more for my specific case. The offending files have the same name within different folders in the work directory. An example of the file structure I am seeing would be:

|-work
   |--3a
      |--82845aa9f7c8a2e485c011c0bb4a5d
         |--bwa
         |--0004.Tumor1-L3_1.fastp.fastq.gz
         |--0004.Tumor1-L3_2.fastp.fastq.gz
         |--Tumor1-L3.0004.bam
   |--5e
      |--f98724d17e629dbde43af318021266
         |--bwa
         |--0004.Tumor1-L3_1.fastp.fastq.gz
         |--0004.Tumor1-L3_2.fastp.fastq.gz
         |--Tumor1-L3.0004.bam

Comparing these bam files via cmp or md5 reveals that they are different files, even after converting them to sam format first.

It might be worth noting that the pipeline failed multiple times due to errors pulling singularity images or errors with processes exceeding running time limits and was run again using the -resume flag. This cycle continued about 10 times, so I’m not sure if the duplicate folders are due to an error in caching the pipeline progress and resuming or if this occurred during the original fastq splitting process in FASTP.

thanks for investigating @brandon-hastings . Wuold you be able to try and reproduce this with a completely clean work directory so we can see whether or not the work directory structure comes from cached steps?

Yes I am running the workflow again now from the beginning with a clean working directory and I can check for the duplicate directory structure if it crashes.

I have previously replicated the behavior where multiple crashes and resumes resulted in the work directory structure I presented above, one in version 3.4.2 and the other in version 3.4.3, both of which were started from the beginning of the Sarek workflow with a clean work directory.

I found that the error was caused by the naming in my sample sheet, which I have included a minimum example of as a txt file. I had unique patient IDs, but was reusing naming for sample IDs across patients which I believe led to the file naming error I saw during the FASTP split because the bam file is named using only the sample ID and lane.

samplesheet:
samplesheet_example.txt

I managed to solve it by manually adding the patient name to the beginning of each sample ID and restarting the pipeline.

Ah interesting. thanks for investigating @brandon-hastings . I will mark this issue with the label input validation. We should add an additional validation step that makes sure sampleIDs themselves are unique for different patients. They still need to be the same within a single patient to account for multiple lanes

I just forked it to take a look for myself and I should be able to submit a pull request to include input validation regarding this issue with an updated subworkflow test sometime over the next few days.