EVA-3571 New stats calculator #196
Conversation
| private int chunkSize; | ||
| private String studyId; | ||
|
|
||
| private static Map<String, Integer> filesIdNumberOfSamplesMap = new HashMap<>(); |
There was a problem hiding this comment.
| private static Map<String, Integer> filesIdNumberOfSamplesMap = new HashMap<>(); | |
| \\ Store the map of files to number of sample from the file_2_0 collection | |
| private static Map<String, Integer> filesIdNumberOfSamplesMap = new HashMap<>(); |
| } | ||
|
|
||
| private MongoCursor<Document> initializeCursor() { | ||
| Bson query = Filters.elemMatch(VariantDocument.FILES_FIELD, Filters.eq(VariantSourceEntryMongo.STUDYID_FIELD, studyId)); |
There was a problem hiding this comment.
I was looking at the indexes in the variant warehouse and there is no index on the StudyId alone.
Instead there is one on study + analysis.
db.variants_2_0.getIndexes()
...
{
v: 2,
key: { 'files.sid': 1, 'files.fid': 1 },
name: 'files.sid_1_files.fid_1',
ns: 'eva_cfamiliaris_31.variants_2_0'
},
...
I thought selecting by study could potentially be slower than retrieving by study+analysis but it turns out that compound indexes support search by the firsts elements of the index as well as the whole.
| import java.util.Set; | ||
| import java.util.stream.Collectors; | ||
|
|
||
| public class VariantStatsProcessor implements ItemProcessor<VariantDocument, VariantDocument> { |
There was a problem hiding this comment.
How would this class work if no genotypes are present ?
There was a problem hiding this comment.
In case any of the variant does not have the field "files" or if the field "files" is empty, the variant will not be processed and the variant will be returned as it is.
Same will happen in case any object in the files array does not have sample data, calculation of stats for that object will be skipped.
| Update update = new Update(); | ||
| update.set("st", variant.getVariantStatsMongo()); |
There was a problem hiding this comment.
Potential optimisation: Compare the stats present before with the result of calculation and only update if the two are different.
Maybe a in a different ticket.
There was a problem hiding this comment.
In light of our discussion this morning, the approach we take will depend on how we decide to handle objects in the files array that share the same sid and fid. Specifically, whether we plan to merge these objects or keep them separate.
If we decide to keep them separate, it will become challenging to accurately map which stats belong to which files object, especially since multiple objects might have the same sid and fid
No description provided.