Storage Nodes API

[rosogon]

[revisions of this spec in https://gist.github.com/rosogon/fb4e6cddcec1b9063903c00d500f86cb]

AGILE will provide a feature to upload data to a storage service or to download sensor data from a storage service.

Storage services are Google Drive, Dropbox, Owncloud.

After a quick search, it looks like these services do not provide an append() operation (GDrive had it in a now discontinued version of the API). Simulating an append() implies downloading a file, appending and re-uploading. For this reason, uploading will always overwrite a file.

There are two use cases:

• Nodered workflows. A nodered node per service will be implemented, which will upload a file stored locally (f.e., by the same workflow). This way, the developer maintains control of what is uploaded and the structure.
• Cloud Storage UI. The user can decide to store (or load) a set of existing sensor data into a service.

NODERed nodes

Existent nodes can be used. For these nodes, the ideal way to configure the credentials is to pass them in the incoming message. See below.

A non-existent node must comply with the following. The configuration values and input MUST be clearly documented in the node.

Configuration

These are the configuration values:

• remotepath: path of file from root folder (e.g. data/sensors/temperature.csv). Parent folders MUST be created if do not exist.
• localpath: path of file to upload (e.g. /tmp/data0001/temperature.csv)
• credentials (takes precedence over msg.credentials)

Input

The input message triggers the upload.

The IDMNode will inject the credentials in the message (msg.credentials field), so the IDMNode will be placed in series.

The node MAY take into account the following values from the input message in case the configuration values are not provided:

• msg.remotepath
• msg.localpath
• msg.payload. If set, the node can optionally use this payload as content to upload, instead of the path specified by localpath.

To summarize:

• remotepath = config.remotepath || msg.remotepath || error
• payload = read(config.localpath) || msg.payload || read(msg.localpath) || error

Output

The node is a regular output node and MUST NOT provide input to other nodes. This way, it behaves as the stock File node.

Cloud Storage UI

Through the AGILE Storage UI, the user can select the data from the local storage (i.e. select the sensing device and the timeframe of stored data) and select one of the supported storage services. Authenticate through the service if needed and push the data through the click of a button.

Upon successful data transfer the user is informed and provided with additional information (like public URL, etc.) where applicable

Form parameters:

• device: to be filled from known devices, if possible.
• component: to be filled from selected sensor, if possible
• from: date/time value
• to: date/time value
• remote path: target path. The file will be named sth like ${gatewayid}${sensorid}${timeinterval}, but the user can edit it.

[from, to) defines the range of timestamps of sensor data to upload.

UI backend layer

The common UI backend layer receives the form parameters. In case of storing to cloud, it has to load the appropriate values from local storage (determined by (device, component, from, to) and send the array of RecordObject to the Data Layer.

Data layer

There will be a data layer for each service in order to provide a uniform interface to Cloud Storage UI.

Store
URL store(Object credentials, byte[] data, String path)

Store content in the service. credentials is a provider-dependent object to authenticate to the service (OAuth2 token, probably). path is the path to a file in the provider (e.g. /agile/temperature). Existing files MUST be overwritten. Intermediate folders MUST be created if needed.

Load
byte[] load(Object credentials, String filename):

Returns the content of a file in the storage service.
credentials is a provider-dependent object to authenticate to the service. path is the path to a file in the provider (e.g. /agile/temperature).

Dataset

Sensor data is in RecordObject format.

Data will be stored in time series, i.e. in collections of data points, which are <timestamp,value> tuples.

The format of a remote file is the json serialization of the RecordObject array, i.e.,

{ 
  "values": [
    {
      "deviceID": "bleA0E6F8B62304",
      "componentID": "Temperature",
      "value": "23.46875",
      "unit": "Degree celsius",
      "format": "",
      "lastUpdate": "1477491668082"
    },
    {...},
    ...
  ]
}

[brettdaman]

The node MAY override the configurations values using the following values from the input message:

msg.remotepath
msg.localpath
msg.payload. If set, the node can optionally use this payload as content to upload, instead of the path specified by localpath.

-> What should be the file name if msg.payload is available?
-> If a localpath is available I use the local file name. Google does not have a problem with duplicate names.
-> msg.remotepath is configurable in my node but not by msg input (1 folder deep from root). it not trivial to upload a file to a specific folder in google drive. This requires mapping between names, parents and folder Ids.

[rosogon]

The node MAY override the configurations values using the following values from the input message:

@brettdaman : First of all, I recently was aware that it should be the opposite. To avoid confusion to developers, if a node parameter is set, it takes precedence over the input msg. If none is supplied, it is an error. I will change the spec to reflect this.

To answer your questions:

-> What should be the file name if msg.payload is available?

remotepath (take first config, then input) is mandatory. So, if payload is available, the filename in server is remotepath

-> If a localpath is available I use the local file name. Google does not have a problem with duplicate names.

This allows some flexibility. I don't have nothing against it, while you clearly document it in the node description.

-> msg.remotepath is configurable in my node but not by msg input (1 folder deep from root). it not trivial to upload a file to a specific folder in google drive. This requires mapping between names, parents and folder Ids.

It's ok to have one folder deep from root. Just document it. It is a next steps item.

remotepath only by parameter. It's ok.

[brettdaman]

-> What should be the file name if msg.payload is available?

remotepath (take first config, then input) is mandatory. So, if payload is available, the filename in server is remotepath

Since I cant do anything to the remote path other than the folder selection in config I will use remotefilename. It is required if the payload is available. I will document this.

Any reason why the msg variables are not camel-cased?

[rosogon]

@brettdaman

Maybe we have a misunderstanding. With remotepath and mean the whole path for the dest file (folder + filename). You can split this parameter into folder and filename parameters (a select box with the possible folders is very nice).

You can also contact me by slackif you have more questions.

[craigmulligan]

@rosogon,

I don't think we should be using msg.localpath to find the file, most of these will be in a flow with the node-red file node as an input and it outputs this:

msg = {
 filename: '/full/path/of/file.txt'
 payload: 'hello world I am content'
}

If we instead use the standard file node as base I think our output of what is uploaded to dropbox would look something like this.

content = msg.payload || msg.filename (read file by path)
filename = config.remotefilename || msg.filename

I think this will reduce the need for functions to normalize the msg before passing it to our cloud storage nodes. What do you think?

[rosogon]

@craig-mulligan,

Thanks for the suggestion. You are right. Please, proceed considering there is a input-file node.

However, I don't fully understand. What parameters are in the configuration of the node?

[craigmulligan]

However, I don't fully understand. What parameters are in the configuration of the node?

The configuration of the node should have whatever credentials you need, eg access_token, api_key, it should also allow you to give a default filename for the node eg remoteFilename = '/myDropboxfolder/syncedFile.txt.

[rosogon]

@craig-mulligan ,

So:

config:

remotepath: path of file from root folder (e.g. data/sensors/temperature.csv). Parent folders MUST be created if do not exist.
credentials (takes precedence over msg.credentials)

input:

msg.remotepath
msg.payload. 

To summarize:

• remotepath = config.remotepath || msg.remotepath || error
• payload = msg.payload

Is it ok?

[craigmulligan]

@rosogon I think it should be:

remotepath = config.filename || msg.filename || error
payload = msg.payload

This is how many of the official node-red nodes work see: https://github.com/node-red/node-red-web-nodes

[rosogon]

@craig-mulligan : Ok

[brett-rombit]

@craig-mulligan a filename and path or not the same.
In my owncloud node i use

content = payload || localpath.fileRead || error
remotepath = msg.remotepath || config.remotepath || error
filename = remotepath.basename(remotepath)

[rosogon]

@brett-rombit ,

As advanced in #5 (comment), to avoid confusions we must prioritize configuration parameters over msg parameters. So, for your owncloud node:

remotepath = config.remotepath || msg.remotepath || error