Output OG 1.0

[callumrollo]

Ultimately, we should aim for the OceanGliders 1.0 format https://github.com/OceanGlidersCommunity/OG-format-user-manual

This will take some time. If you spot an atomic change that can be made to move toward OG 1.0, please submit an Issue for it

[elyfant]

Hi Callum, I'm currently looking into outputing in OG1.0 format. Do you know if any work have been done on this so far in pyglider?

[jklymak]

I haven't tried yet, but it would be good to figure out what is currently not supported. I'd think og1.0 is 95% supported already just someone needs to make a yaml example.

[elyfant]

ok, thanks jody. will have a look

[richardsc]

Hi all! @clayton33 has made some progress with code (outside of pyglider) to convert the pyglider nc files to an OG1.0 compliant version. We are still wrapping up our processing pipeline, and weren't sure if this is something that should propagate back into pyglider or not, but this might be a good time to discuss.

@clayton33 -- can you provide a brief overview of how you're handling this at the moment?

[jklymak]

Is there a reason to not just modify the yaml to have the OG1.0 compatible names and metadata? It should work fine. OG1.0 also has quality flags, which we do not populate, but maybe would be a simple addition.

[richardsc]

@clayton33 can correct me on this, but the reason we converted them after was to have both available (as we are still trying to sort out exactly where the files are going to end up and what format will be required).

But yes, modifying the yaml should also work, I believe.

[clayton33]

From my experience so far, nothing needs to be changed in pyglider to make files OG1.0 compliant. In order to make the files OG1.0 compliant, variable names and additional metadata needs to be indicated in the .yaml file. It's fairly simple for non-IOOS variables, for example, in the .yaml file, backscatter could look something like below

  BBP700:
    coordinates: time depth latitude longitude
    instrument: instrument_fluorometer
    long_name: Particle backscattering at 700 nanometers
    observation_type: measured
    source: FLBBCD_BB_700_SCALED
    standard_name: ''
    units: /m/sr
    unitsvocabulary: http://vocab.nerc.ac.uk/collection/P06/current/PMSR/
    vocabulary: http://vocab.nerc.ac.uk/collection/OG1/current/BBP700/

but for IOOS variables, the main ones being pressure, temperature, conductivity, and salinity, you'll have to change this after it's processed. As an example, temperature in the .yaml file looks like the below,

  temperature:
    coordinates: time depth latitude longitude
    instrument: instrument_ctd
    long_name: Sea temperature in-situ ITS-90 scale
    observation_type: measured
    replaceName: TEMP
    source: GPCTD_TEMPERATURE
    standard_name: sea_water_temperature
    units: degC
    unitsvocabulary: http://vocab.nerc.ac.uk/collection/P06/current/UPAA/
    vocabulary: http://vocab.nerc.ac.uk/collection/OG1/current/TEMP/

you'll see I have something called replaceName, which in this case is the OG1.0 variable name. Once the files are processed, I have a post processing script which will change the name of the variable in the .nc file to the replaceName.

[jklymak]

If you are willing it might be nice to share that utility via pyglider.

I think the todos here are

1. make a fully OG1.0 compliant yaml file example(s) to ship w/ pyglider. Note for multiple formats one could easily just run your script twice with different yaml files.

2. write a translator. This should be pretty straightforward and quick if all the data is in the original file. It would be nice if it also ran off a yaml file, maybe something like:

TEMP:
    origname: temperature
    renamed_attr:
         attr_name1_old: attr_name1_new
         attr_name2_old: attr_name2_new
   copied_attr:
         attr_name3_old : attr_name21_new
   dropped_attr:
         attr_name4_old
         attr_name5_old
   newattr1: attr1_value

Not sure how much attribute renaming/copying would have to happen, but could be useful in other contexts..

[callumrollo]

We've been working with a similar process to @clayton33 to post-process our files into OG1 format. It would be good to upstream these changes into pyglider though. I'll work on making a compliant yaml file and example to ship with pyglider as @jklymak suggests.

I'm not sure all the things OG1 requires can be achieved with yaml, but I'll try to minimise changes to the codebase

[jr3cermak]

I am slowly shoring up a data pipeline for UAF for glider processing using pyglider. One of these tasks is to ensure compliance with as much of the OG-1.0 checker as possible. I did this with a different pipeline already. I would have to port all this over to the slocum.py driver and as well define that in yaml. Anyway, just saying we also have a parallel track that will hopefully converge with everyone else. I just shored up pyglider to at least be compliant with cf-1.8 and ngdac 2.0. Just need to be sure things are working at UAF before sending additional PRs here. Refreshing the existing PR #163 for compliance.

[jklymak]

Just out of curiosity, are any glider DACs ingesting OG1.0 yet?

[richardsc]

We're about to start pushing DFO glider data to CIOOS, and I'm going to insist that they ingest OG1.0 😄