I/O Plugin Architecture

[fluffynukeit]

What is the status of the I/O plugin architecture developments listed in this paper from circa 2013?

https://library.psfc.mit.edu/catalog/reports/2010/13ja/13ja022/13ja022_full.pdf

Particularly:

Each plugin can replace treeshr routines at different levels of its call stack. Some like the
mdsip plugin replace the atomic file input / outout routines. Others like the mongodb one, which
is under development, replace higher level treeshr calls such as TreeOpen, TreeGetRecord, or
TreeGetNci. Mongodb is an open source NOSQL database. It provides tables with
heterogenous rows, and includes replication, sharding and network access[4]. The requirement
is that they implement a functionally complete set of routines..

[joshStillerman]

Hi
We have been thinking a lot about this recently, and are restarting on some prototypes. There are several approaches, each with there own limitations and benefits. Current thoughts:

• MDSplus expression evaluator performance is correlated with the speed at which serial transactions can be performed. To answer 'a + b' we first evaluate 'a' and then evaluate 'b', if 'a' and or 'b' have many steps, they have to be performed serially.
  • This could possibly be mitigated by parallel IO
  • This could possibly be mitigated by record level caching
  • This could possibly be mitigated by answer level caching
  • (These could also be combined)
• We could replace treeshr, and make it so that different treeshr images could be used
• We could modify existing treeshr to distribute the IO
• We could replace the IO routines of treeshr

This will all be discussed and a plan moving forward will be developed during the upcoming (January) MDSplus developers meeting.

We now have concerns about the longevity, durability and scalability of Mongo DB, but it is one of the contenders. We are also thinking of using something like redis to cache information that we need fast access to,

[fluffynukeit]

As a user, I am most interested in using alternative data storage backends. MongoDB would be one option. Columnar Postgresql is another, which might better support some operations. Another is simply putting the data into the file system hierarchy in a suitable way. But in any case, the goal would be to better disassociate the shot tree data model from the data storage format so that deployments could elect to use the data storage technology that makes the most sense while still being compatible with the MDSplus ecosystem (e.g. traverser, jscope, MDSplus Python module, etc).

[GabrieleManduchi]

Maybe the most straightforward solution is replacing treeshr library. Treeshr defines a number of routines and represents the unique access point to data storage in MDSplus, So, if treeshr is replaced by any other implementation with the same interface, all the MDSplus ecosystem can be retained.
This is for sure not the optimal solution and a better plugin interface could be defined (this has been already done for example in mdsip in order use communication protocols different from TCP/IP), but in this way no change in current MDSplus code is required.
In conclusion, if you are willing to investigare alternative storage strategies and implement a prototype, I suggest starting from treeshr re-implementation. A desirable side effect of this work would be the definition of a plugin interface that can be agreed for using I/O plugins, based on the experience gained in such a prototype develiopment.