add .ssf for many packages

[93Boy]

Adding ENA data tables to current packages

[nevrome]

Great! Here are some things I think should be changed:

• Please remove all .py files. The scripts are not supposed to be in the package library -> Dana.
• Please reference the newly added ena tables in the respective POSEIDON.yml files as defined in the upcoming poseidon schema release (also including a checksum). Please make sure the fields are named correctly (sequencingSourceFile) -> Dana.
• Please remove superfluous quotes from the .janno files where you accidentally added them. We should not change file checksums if there is no need. -> Thiseas will provide sed-magic. Dana will do it.
• Column names in SSF files need to be adapted to lower-case versions. Please check the schema of the sequencingSourceFile. -> Thiseas can provide sed, please, and Dana may use it.

I think we could also remove the quotes from the ena table .tsv files, if they're not necessary. In libre office, for example, you can specify the writing behaviour for .csv/.tsv files and turn off wrapping everything in quotes.

[TCLamnidis]

Some ENA tables have almost no quotes (e.g. 2020_AgranatTamir_LevantBA/ENAtable.tsv), while others have quotes in every field (e.g. 2020_Nagele_Caribbean/ENAtable.tsv).

Why is that?

[TCLamnidis]

Lines 1626-1673 look strange. (Github seems to preview different ones 😕 )
sample_alias looks like ena-SAMPLE-TAB-28-10-2021-18:52:59:284-23886, and Poseidon_ID field is empty

[93Boy]

I was unable to match 2020_AgranatTamir_LevantBA package with Poseidon data as it has completely different namings. I have opened this draft PR #116 for this

[TCLamnidis]

Re: 2021_PattersonNature
I manually curated the SSF file for this package for my own uses, which you could copy over so we avoid doing work twice. I took the table you created and, using the paper's supplementary table as well as the AADR, filled in the library_built and udg columns. I also fixed the sample_alias and poseidon_IDs for the lines that I complained about above, using the BAM name as the IID.
I have added some markings to the sample_accession (see this PR for explanations). These will be removed after we we discuss things in our next meeting, after which the SSF should be valid and can be copied over.

For future reference, you can find the updated file here. 😃

[stschiff]

Right, I think the quotes were also mentioned by Clemens in some of the Janno Tables. @93Boy do you know how to take it from here? Or shell we help with some scripting to get this PR in order?

[TCLamnidis]

This sed command should remove any double quotes from a given file.

sed -e 's/"//g' file.ssf >updated_file.ssf

You can edit the files in place with the following command instead (no new file created):

sed -i -e 's/"//g' file.ssf

[nevrome]

After a request for feedback here's what I see:

• There is still one .py file in 2018_VeeramahPNAS.
• The field names in the POSEIDON.yml files still have to be changed: enatableFile -> sequencingSourceFile, enatableFileChkSum/.enatableFileChkSum -> sequencingSourceFileChkSum. Please see the schema definition here.
• There are superfluous, empty lines in the beginning of the POSEIDON.yml files. While they shouldn't be a problem I think it would be better to remove them. But if the validation doesn't complain, then I don't care too much.
• The column poseidon_id in the .ssf files should be called poseidon_IDs. Please see the schema definition here.

It's hard to systematically check the .ssf files. Good that we have the automatic validation to test everything in the end. For now I saw the following issues in some random files I inspected:

• In 2019_Nikitin_LBK/ENAtable.ssf the columns udg and library_built are swapped. UDG treatment is documented in the strandedness column and vice versa.
• In the same file I saw the entry mixed in the udg column. This entry is allowed in .janno files, but not in .ssf files, where the column can only have one of the values minus;half;plus.

The first issue I observed in at least one more file. The automatic validation will highlight all files with these two issues. What it will not show are the cases where one or both of these columns are missing. But that is fine.

Beyond that I saw the NA value #N/A in at least one .ssf file (2020_Marcus_Sardinia/ENAtable.ssf). I don't like that, but I admit that we didn't specify an NA value for .ssf files. I think we should just stick to what we use for the .janno files: n/a or empty strings.

[93Boy]

@nevrome I have made all the changes except the mixed udg treatment concern. What should be the substitute for mixed values?

[nevrome]

Great! Almost all ToDos I saw above are resolved. What I still see:

• A merge conflict with 2020_Nakatsuka_SouthPatagonia/Nakatsuka_SouthPatagonia.janno, which has to be resolved. Can be done with the Resolve conflicts feature here on GitHub.
• The remnants of an unresolved merge conflict in 2021_CarlhoffNature/ENAtable.ssf. Search for <<<<<<< HEAD to find it. I guess that has something to do with @TCLamnidis doing.
• 2020_Nakatsuka_SouthPatagonia/Nakatsuka_SouthPatagonia.janno and 2021_PattersonNature/2021_PattersonNature.janno got some unnecessary quotes when the Genetic_Source_Accession_IDs were added (I assume). They should ideally be removed again. Also in 2019_Nikitin_LBK/ENAtable.ssf.

@dhananjaya93: As this keeps coming up: Maybe you could try setting Quote all text cells to False in the LibreOffice Calc .csv saving dialogue, after selecting Edit filter settings when you set the output name and format. Afaik this option stays like that after setting it once.

Unlike the other packages, 2021_Zegarac_SoutheasternEurope already has a new entry in the CHANGELOG file. Be careful with that, when you apply update to everything in the end.

Regarding the mixed in the udg column: I think in this case it's OK to leave the column empty for now, right, @TCLamnidis? This is information that can not be easily obtained from the data we have atm and has to be added later.

[stschiff]

OK, good job @93Boy @dhananjaya93 (which one is it now?) resolving most. I think we're happy to help resolving these last outstanding points, unless you want to resolve them yourself. Please let us know briefly whether you're working on it, otherwise I suggest that someone else resolves these quickly so we can finally merge this.

[93Boy]

Yes I really appreciate if someone kindly fix these issues. So I can focus on adding rest of the ENA information quickly and complete this task asap. For the future packages, I will set Quote all text cells into fales as @nevrome mentioned. btw I am using only @93Boy account for the commits

[nevrome]

I added a warning to trident to identify .ssf files with no poseidon_IDs columns/entries. Here's what it returns:

[Warning] Potential consistency issues in file ./2019_BraceNatureEcologyEvolution/ENAtable.ssf: The poseidon_IDs column is completely empty. Package and .ssf file are not linked
[Warning] Potential consistency issues in file ./2020_Bergstrom_HGDP/ENAtable.ssf: The poseidon_IDs column is completely empty. Package and .ssf file are not linked
[Warning] Potential consistency issues in file ./2019_Haber_Crusaders/ENAtable.ssf: The poseidon_IDs column is completely empty. Package and .ssf file are not linked
[Warning] Potential consistency issues in file ./2019_Jeong_InnerEurasia/ENAtable.ssf: The poseidon_IDs column is completely empty. Package and .ssf file are not linked
[Warning] Potential consistency issues in file ./2019_Brace_Britain/ENAtable.ssf: The poseidon_IDs column is completely empty. Package and .ssf file are not linked
[Warning] Potential consistency issues in file ./2021_Yaka_Anatolia/ENAtable.ssf: The poseidon_IDs column is completely empty. Package and .ssf file are not linked

This renders these .ssf files a bit pointless, as already mentioned by @TCLamnidis in our discussion today. Not easy to fix, though, so maybe we should just remove them for this PR. Then @93Boy can work them in with another PR later.

[nevrome]

I did this and removed the pointless .ssf files here. They live now in #118, #119, #120, #121, #122 and fall under your responsibility, @93Boy 👍

[stschiff]

OK, I have looked briefly into the directory structure of this PR and some coarse look at some SSFs. What's the status regarding the automatic checking? I am happy to approve this PR in principle!

[nevrome]

Automatic checking should be done with a trident version from this PR: poseidon-framework/poseidon-hs#245
For me this validation passes alright now.