Provide reference datasets

[gavinmacaulay]

We need reference datasets for validation of code that reads and writes files in the convention format. Preferably files that are small enough to store in this repository (if not, we can arrange an online place to store the larger files). Files should be accompanied by a small description of their contents. Any files provided should have no distribution restrictions on them.

(Suggested by @lberger29)

[lberger29]

A reference dataset for EK60 and ME70 data is available at https://www.seanoe.org/data/00475/58652/

[geoffmatt]

This is a good idea, but some care should be taken here.
a) It would be good to have some reference for what the data should look like when loaded. Possibly in another format like a Simrad one so that it can be loaded in both formats and compared.
b) @lberger29 - Laurent, I'm not sure your dataset is compliant. The backscatter_r and backscatter_i variables are 3-dimensional in those files, whereas the CRR specifies that they should be 2D. Because of this Echoview will not read these files at the moment

[lberger29]

a) The corresponding Simrad proprietary raw files have been made available at the same location for the reference dataset (https://doi.org/10.17882/58652)
b) The subbeam dimension allows to store the raw data of each subarray for a split-beam transducers in the current version of the convention and is an optional dimension. The dataset provided is however compliant with a subbeam dimension equal to one and this dimension is needed for EK80 Sonar-Netcdf data files with complex samples.

[geoffmatt]

a) Yep, my mistake, I can see the Simrad files in that location, sorry about that
b) I know I'm sounding pedantic, because the subbeams have been added to the development version of the format, but they are not in the currently published version. Until V2.0 is published we cannot consider it the "current version", because we cannot code support for a format which is changing on a daily basis. This is a very frustrating problem for us at Echoview.
If these files are to be considered "reference" they need to be associated with a specific version of the format, otherwise they cannot be reliably utilised as a reference. Seeing as they have been created against the current iteration of V2, they should be associated with an exact version, i.e. 2.0.13.
This is important because V2 is likely to change again before publication, in which case these files may not be compliant with that newer V2.

[lberger29]

I agree Geoffroy that we need to publish a version 2.0 of the convention.
I want to add some adiditionnal variables for properly handling filters and pulse shape in the beamgroup but this can be in version 2.1
@gavinmacaulay Gavin, do you think that we are able to publish a version 2.0 of the online convention?

[cyrilleponcelet]

I agree, we should go as soon as possible to a V2.0 release.
I would prefer to go through a kind of final review process planning something like :

• notify and people that a v2.0 is coming, we request people to send their pending request or changes they want to be integrated in this release (like It is not possible to query the ping sample count, or range limits, without reading the entire ping #28 or Add to Platform description an explicit reference to CF single trajectory representation #56) , reviewers can decide if this is for 2.1 or 2.0 release. We could give a few weeks for people to react.
• and then a few weeks for a final review where we only accept corrections or clarifications.

This could lead to a final release somewhere in december.

[gavinmacaulay]

I agree on Cyrille's plan and suggest we turn this into a procedure document that we follow for this and future releases - I can make a PR for this

[gavinmacaulay]

Please look at PR #59