Reference examples of components that generate the input binaries from csv have been provided. One outstanding component to convert cdf data from csv to binary will be provided.
The input data for the reference components eve and getmodel can be split across several files, where each is identified by a chunk_id under a fixed naming convention. Eve and getmodel have 'chunk_id' as an input parameter which identifies the relevant input binary file.
Because chunk is an Oasis mid-tier concept, the chunk parameter will be removed and each internal data input to the ktools reference model will be a single consolidated file.
Note that the events can still be chunked and each chunk distributed to a separate back-end running the in-memory kernel.
Currently if the results of the intermediate calculation steps are required to be persisted, then they must be written out to disk and read back into memory to continue downstream processing.
The plan is to enable intermediate calculation steps to be written out to disk whilst continuing the in-memory workflow.
The sample index field is currently a 4 byte integer format for simplicity of design but we estimate gulcalc stdout data volumes could be reduced by approximately 25% if we reduce it to 2 bytes. This would mean a maximum number of samples of 64,000.
There is a fixed limit of 1 million random numbers per event when the dynamic random number option (no -r parameter) is used in gulcalc. This limit was imposed to improve performance. We plan to add this limit as an optional parameter to gulcalc.
The fmcalc code will be re-factored to improve efficiency, performance and extensibility.