Error during functional profiling

[arfaiq]

Hello,

I am getting the following error for functional profiling when running the pipeline (this is the log):

diamond v2.0.15.153 (C) Max Planck Society for the Advancement of Science
Documentation, support and updates available at http://www.diamondsearch.org
Please cite: http://dx.doi.org/10.1038/s41592-021-01101-x Nature Methods (2021)

#CPU threads: 80
Scoring parameters: (Matrix=BLOSUM62 Lambda=0.267 K=0.041 Penalties=11/1)
Temporary directory: /cluster/projects/nn9383k/arfa/funomic_analysis/output_folder/functional_profiling
#Target sequences to report alignments for: 25
Opening the database... [0.447s]
Database: /cluster/projects/nn9383k/arfa/funomic_analysis/FunOMIC.P.v1//FunOMIC.P.v1.dmnd (type: Diamond database, sequences: 3413239, letters: 1879525586)
Block size = 5000000000
Opening the input file... [0.001s]
Error: Error detecting input file format. First line seems to be blank.

[ManichanhLab]

Hi arfaiq,

Could you please send me the content of your input file by doing this:
zcat /cluster/projects/nn9383k/arfa/funomic_analysis/output_folder/yourFastq.gzFile | head

[arfaiq]

Hi,

This is what I get:

$ zcat mock_fungal_paired1.fq.gz File | head

@GCF_000002515.2_ASM251v1_genomic-2000000/1
GAAAAAAGTGCTCAAGAAGTTATCGAGGAAAAAAATACCAAATTATTCGGTGCTATTCTAACTGGTTTGAATCGTGCTTTCCCATTCTCTGATATGCCGGGATCCGTGTACGAGATACATCTTGATACATTGTTCAAGATAACACATGCG
+
CCC1GGGGGGGGGJJGJJJJJJGJJ=JJJJJGCGGJJJGGGJJJGJJJJGJJCJGJJGGJJGJJJ=JGG8JJGGGGGGGG8CCC=GG8CCGGGGGGGGGGJCCCCGGGGGGGGCGG=GGCGG=GCGCGGGCCCGGCGGGGCGCG1GG=GC
@GCF_000002515.2_ASM251v1_genomic-1999998/1
GCAGAGAAACAAGAACCGGCCATGGAAATGCATTACATTCAGTGGCAATTTGTAGGACCTAAAACCGTCCACTGTATTATGACTACTCTAGCAGAGTATAAATTGCACAAAGAATTTGCAAGACCTCATACTACTCTACAGTTCCACTTG
+
C=CGGGGGGGGGGJGJJ=JJGJJJJGJJGJG=JJJGJG=JJGCGCGGJGJJGJJJGCJCCGGGG=GG=GGGGJGC8GG1CGGGCGGGGCG=GGCCG1GGGJ8GCGGGGCGGGCCG=GGGCG81CGCGGGCCCCGGGG1GGG=GGCGGCCG
@GCF_000002515.2_ASM251v1_genomic-1999996/1
ACACTTTTGGTAAAGTTGCGGGAATTGAAGACAAAATGCTGACAGAGAGACTTAAATGATTGCCAGTGGTAACGACATTTCATAATAACATTGACACCTGATTCGTATACAAAGCAGACTAAGGTCTTCTGAACTGGAGAATTGGATGCT

[MarwanMaj]

hi I am having similar issue to the above

Also this is the error log when submitting the job

FunOMIC.sh -1 /srv/scratch/z3192995/LOTUS_meta/READ_QC_1/MTG220_a/final_pure_reads_1.fastq -2 /srv/scratch/z3192995/LOTUS_meta/READ_QC_1/MTG220_a/final_pure_reads_2.fastq -p MTG220_a -o /srv/scratch/z3192995/LOTUS_meta/funomic/MTG220 -a /srv/scratch/z3192995/DB/funomic/New_folder/BacterialDB -b /srv/scratch/z3192995/DB/funomic/New_folder/FunOMIC_T -c /srv/scratch/z3192995/DB/funomic/New_folder/FunOMIC.P.v1 -t 8
Start removing bacterial reads for MTG220_a
[W::sam_hdr_create] Ignored @sq SN:kr : bad or missing LN tag
[E::sam_hrecs_update_hashes] Header includes @sq line "kr" with no LN: tag
samtools view: failed to add PG line to the header
Start taxonomic annotation for MTG220_a
[W::sam_hdr_create] Ignored @sq SN:Zyghe : bad or missing LN tag
[E::sam_hrecs_update_hashes] Header includes @sq line "Zyghe" with no LN: tag
samtools view: failed to add PG line to the header
[W::sam_hdr_create] Ignored @sq SN:Zyghe : bad or missing LN tag
[E::sam_hrecs_update_hashes] Header includes @sq line "Zyghe" with no LN: tag
samtools view: failed to add PG line to the header
[E::hts_open_format] Failed to open file "/srv/scratch/z3192995/LOTUS_meta/funomic/MTG220/tmp/MTG220_a.30.sorted.bam" : No such file or directory
samtools view: failed to open "/srv/scratch/z3192995/LOTUS_meta/funomic/MTG220/tmp/MTG220_a.30.sorted.bam" for reading: No such file or directory
grep: /srv/scratch/z3192995/LOTUS_meta/funomic/MTG220/tmp/MTG220_a.30.c80.list: No such file or directory
[E::hts_open_format] Failed to open file "/srv/scratch/z3192995/LOTUS_meta/funomic/MTG220/tmp/MTG220_a.30.c80.bam" : No such file or directory
samtools idxstats: failed to open "/srv/scratch/z3192995/LOTUS_meta/funomic/MTG220/tmp/MTG220_a.30.c80.bam": No such file or directory
Starting functional annotation for MTG220_a
merged clean reads stored in: /srv/scratch/z3192995/LOTUS_meta/funomic/MTG220/functional_profiling/joined.fastq
awk: fatal: cannot open file `/srv/scratch/z3192995/LOTUS_meta/funomic/MTG220/functional_profiling/MTG220_a.func.out' for reading (No such file or directory)

[MarwanMaj]

hi I am having similar issue to the above

Also this is the error log when submitting the job

FunOMIC.sh -1 /srv/scratch/z3192995/LOTUS_meta/READ_QC_1/MTG220_a/final_pure_reads_1.fastq -2 /srv/scratch/z3192995/LOTUS_meta/READ_QC_1/MTG220_a/final_pure_reads_2.fastq -p MTG220_a -o /srv/scratch/z3192995/LOTUS_meta/funomic/MTG220 -a /srv/scratch/z3192995/DB/funomic/New_folder/BacterialDB -b /srv/scratch/z3192995/DB/funomic/New_folder/FunOMIC_T -c /srv/scratch/z3192995/DB/funomic/New_folder/FunOMIC.P.v1 -t 8 Start removing bacterial reads for MTG220_a [W::sam_hdr_create] Ignored @sq SN:kr : bad or missing LN tag [E::sam_hrecs_update_hashes] Header includes @sq line "kr" with no LN: tag samtools view: failed to add PG line to the header Start taxonomic annotation for MTG220_a [W::sam_hdr_create] Ignored @sq SN:Zyghe : bad or missing LN tag [E::sam_hrecs_update_hashes] Header includes @sq line "Zyghe" with no LN: tag samtools view: failed to add PG line to the header [W::sam_hdr_create] Ignored @sq SN:Zyghe : bad or missing LN tag [E::sam_hrecs_update_hashes] Header includes @sq line "Zyghe" with no LN: tag samtools view: failed to add PG line to the header [E::hts_open_format] Failed to open file "/srv/scratch/z3192995/LOTUS_meta/funomic/MTG220/tmp/MTG220_a.30.sorted.bam" : No such file or directory samtools view: failed to open "/srv/scratch/z3192995/LOTUS_meta/funomic/MTG220/tmp/MTG220_a.30.sorted.bam" for reading: No such file or directory grep: /srv/scratch/z3192995/LOTUS_meta/funomic/MTG220/tmp/MTG220_a.30.c80.list: No such file or directory [E::hts_open_format] Failed to open file "/srv/scratch/z3192995/LOTUS_meta/funomic/MTG220/tmp/MTG220_a.30.c80.bam" : No such file or directory samtools idxstats: failed to open "/srv/scratch/z3192995/LOTUS_meta/funomic/MTG220/tmp/MTG220_a.30.c80.bam": No such file or directory Starting functional annotation for MTG220_a merged clean reads stored in: /srv/scratch/z3192995/LOTUS_meta/funomic/MTG220/functional_profiling/joined.fastq awk: fatal: cannot open file `/srv/scratch/z3192995/LOTUS_meta/funomic/MTG220/functional_profiling/MTG220_a.func.out' for reading (No such file or directory)

These are the error logs for each step
Functional_profiling folder log :

diamond v2.0.15.153 (C) Max Planck Society for the Advancement of Science
Documentation, support and updates available at http://www.diamondsearch.org
Please cite: http://dx.doi.org/10.1038/s41592-021-01101-x Nature Methods (2021)

#CPU threads: 4
Scoring parameters: (Matrix=BLOSUM62 Lambda=0.267 K=0.041 Penalties=11/1)
Temporary directory: /srv/scratch/z3192995/LOTUS_meta/funomic/MTG220_a/functional_profiling
#Target sequences to report alignments for: 25
Opening the database... [2.23s]
Database: /srv/scratch/z3192995/DB/funomic/New_folder/FunOMIC.P.v1/FunOMIC.P.v1.dmnd (type: Diamond database, sequences: 3413239, letters: 1879525586)
Block size = 5000000000
Opening the input file... [0.001s]
Error: Error detecting input file format. First line seems to be blank.

Taxonomy_profiling folder:
stat: Bad file descriptor
Warning: Could not open read file "/srv/scratch/z3192995/LOTUS_meta/funomic/MTG220_a/MTG220_a_noBact_1.fastq.gz" for reading; skipping...
Error: No input read files were valid
(ERR): bowtie2-align exited with value 1

bact_decontam.log
(ERR): bowtie2-align died with signal 9 (KILL)

[zixuanxie]

Hi,

This is what I get:

$ zcat mock_fungal_paired1.fq.gz File | head

@GCF_000002515.2_ASM251v1_genomic-2000000/1 GAAAAAAGTGCTCAAGAAGTTATCGAGGAAAAAAATACCAAATTATTCGGTGCTATTCTAACTGGTTTGAATCGTGCTTTCCCATTCTCTGATATGCCGGGATCCGTGTACGAGATACATCTTGATACATTGTTCAAGATAACACATGCG + CCC1GGGGGGGGGJJGJJJJJJGJJ=JJJJJGCGGJJJGGGJJJGJJJJGJJCJGJJGGJJGJJJ=JGG8JJGGGGGGGG8CCC=GG8CCGGGGGGGGGGJCCCCGGGGGGGGCGG=GGCGG=GCGCGGGCCCGGCGGGGCGCG1GG=GC @GCF_000002515.2_ASM251v1_genomic-1999998/1 GCAGAGAAACAAGAACCGGCCATGGAAATGCATTACATTCAGTGGCAATTTGTAGGACCTAAAACCGTCCACTGTATTATGACTACTCTAGCAGAGTATAAATTGCACAAAGAATTTGCAAGACCTCATACTACTCTACAGTTCCACTTG + C=CGGGGGGGGGGJGJJ=JJGJJJJGJJGJG=JJJGJG=JJGCGCGGJGJJGJJJGCJCCGGGG=GG=GGGGJGC8GG1CGGGCGGGGCG=GGCCG1GGGJ8GCGGGGCGGGCCG=GGGCG81CGCGGGCCCCGGGG1GGG=GGCGGCCG @GCF_000002515.2_ASM251v1_genomic-1999996/1 ACACTTTTGGTAAAGTTGCGGGAATTGAAGACAAAATGCTGACAGAGAGACTTAAATGATTGCCAGTGGTAACGACATTTCATAATAACATTGACACCTGATTCGTATACAAAGCAGACTAAGGTCTTCTGAACTGGAGAATTGGATGCT

Hi @arfaiq , sorry I made a mistake. The input for this step should be the merged file ended named "joined.fastq" under the functional_profiling folder. Could you try this code to check if you can generate this file?
flash2 XXX_noBact_1.fastq.gz XXX_2.fastq.gz -o functional_profiling/tmp/XXX -q -t $threads cat functional_profiling/tmp/XXX.extendedFrags.fastq functional_profiling/tmp/XXX.notCombined_1.fastq functional_profiling/tmp/XXX.notCombined_2.fastq > functional_profiling/joined.fastq

If yes, can you send me the first few lines of this file? Thanks!

[arfaiq]

Hi @arfaiq , sorry I made a mistake. The input for this step should be the merged file ended named "joined.fastq" under the functional_profiling folder. Could you try this code to check if you can generate this file? flash2 XXX_noBact_1.fastq.gz XXX_2.fastq.gz -o functional_profiling/tmp/XXX -q -t $threads cat functional_profiling/tmp/XXX.extendedFrags.fastq functional_profiling/tmp/XXX.notCombined_1.fastq functional_profiling/tmp/XXX.notCombined_2.fastq > functional_profiling/joined.fastq

If yes, can you send me the first few lines of this file? Thanks!

Hi, after adding this to the code, I get the following logs:

Taxonomical profiling:
0 reads
0.00% overall alignment rate

Functional profiling:
diamond v2.0.15.153 (C) Max Planck Society for the Advancement of Science
Documentation, support and updates available at http://www.diamondsearch.org
Please cite: http://dx.doi.org/10.1038/s41592-021-01101-x Nature Methods (2021)

#CPU threads: 80
Scoring parameters: (Matrix=BLOSUM62 Lambda=0.267 K=0.041 Penalties=11/1)
Temporary directory: /cluster/projects/nn9383k/arfa/funomic_analysis/output_folder/functional_profiling
#Target sequences to report alignments for: 25
Opening the database... [0.248s]
Database: /cluster/projects/nn9383k/arfa/funomic_analysis/FunOMIC.P.v1//FunOMIC.P.v1.dmnd (type: Diamond database, sequences: 3413239, letters: 1879525586)
Block size = 5000000000
Opening the input file... No such file or directory
[0s]
Error: Error calling stat on file /cluster/projects/nn9383k/arfa/funomic_analysis/output_folder/functional_profiling/joined.fastq

Bacterial decontamination log:
(ERR): bowtie2-align died with signal 11 (SEGV) (core dumped)

This is what the code looks like now:

##############################################
########### FUNCTIONAL PROFILING! ############
##############################################

printf "Starting functional annotation for ${prefix}\n"

cd ${outdir}/functional_profiling
mkdir $outdir/functional_profiling/tmp/
```

```
flash2 $outdir/${prefix}_noBact_1.fastq.gz $outdir/${prefix}_noBact_2.fastq.gz -o tmp/${prefix} -q -t $threads &>/dev/null
cat $outdir/functional_profiling/tmp/${prefix}.extendedFrags.fastq $outdir/functional_profiling/tmp/${prefix}.notCombined_1.fastq $outdir/functional_profiling/tmp/${prefix}.notCombined_2.fastq > /functional_profiling/joined.fastq

```


`printf "merged clean reads stored in: $outdir/functional_profiling/joined.fastq \n"`

[zixuanxie]

Hi @arfaiq

Looks like you have problem at the bacterial reads removal step, is it possible to share your data so that I can test at my side?