Start integrating OBIS datasets within cioos-explore

[JessyBarrette]

CIOOS is also distributing a fair bit of data via OBIS. OBIS is also already standardizing the data and makes it available via single API. This suggest integrating CIOOS related datasets from OBIS could potentially be manageable. The mains issues I could see are:

• This could potentially be a lot of single points records instead of timeseries at a single location. I wonder if CDE could potentially be capable of handling that.
• How do you harvest CIOOS related OBIS datasets? Via the ckan catalogue?
• Should we make the download part compatible with OBIS?
• How do we link CKAN Catalogue datasets to their OBIS equivalent.

[vmehraOCN]

Quick question. Does the current harvester get all available data for a given target and download it into our CDE DB? My concern is duplicating data from OBIS which doesn't really need to be treated as such.

1. We could setup a pagination so that segments of a dataset come up depending on some kind of upper limit, say 1000 rows. Part two of

2. Obis does have an API that is well documented. I think we can use this to leverage our selections of datasets if CKAN catalogue doesn't have what we need.
   Eg. /occurrence/grid/{precision} and all other get methods in the Occurrence, Dataset and quite a few others have Institute ID as a field to filter. We would need to think up of our requirements and confirm if it would work.

3. What would this look like? I was thinking that we could link the obis page to download this info so we don't have to worry about our own implementation and save on system resources. If we already download the the datasets from obis int our own DB then we can just serve from our side with a notice saying this data comes from obis.

4. I'm working on this part. Hopefully I can use the metadata sign up form to sign up an OBIS source into CKAN. This might break harvester or simply not work so I'll have to be think of what to do here.

[JessyBarrette]

Explore-cioos do not store any actual data internally and retrieve it on the client side when a user requests to see a specific time series or profile via the table/plot interface. We limit the view to 1000 points.

explore-cioos only captures the following information for a specific dataset/subset:

• start/end time
• min/max latitude
• min/max longitude
• min/max depth/altitude
• metadata (title, variables)

Only this information is stored within the Explorer database. Everything else is done on the client side.

regarding your points:

1. Since we're not hosting any actual data I don't think we need it.
2. OBIS gives access to its occurrence/grid/ interface this could potentially be useful to use :)
3. I agree that a simple link to the original OBIS page is a good start. We won't host any of the actual data.
4. Ideally your tool would only interact with the CKAN API and would not deal with the metadata form API.

[vmehraOCN]

I'm still trying to understand CKAN and how it ties in, but to my current understanding is that you need to have a completed metadata entry form in order to create a new entity in CKAN. Is there an issue with my understanding of this requirement?

[vmehraOCN]

@JessyBarrette, I found a possible option in helping us add obis resources as a contributor to CKAN via this api call. There is the matter of authentication which I will dig into (maybe a new env variable), but it seems to be the best way we can add another contributor.

From there, I can add to the harvester for obis specific data (api calls for required datasets and the processing needed for it).

[JessyBarrette]

Sorry I forgot to reply here. CKAN is only our frontend tool to distribute metadata. The metadata form is use to generate a new record for an RA, this is as now the only way to generate a new CIOOS dataset. We are not using the CKAN API to add new datasets. Le ven. 14 févr. 2025 à 09:20, Vishal Mehra ***@***.***> a écrit :

…

@JessyBarrette <https://github.com/JessyBarrette>, I found a possible option in helping us add obis resources as a contributor to CKAN via this api <https://docs.ckan.org/en/2.9/api/index.html#module-ckan.logic.action.create> call. There is the matter of authentication which I will dig into (maybe a new env variable), but it seems to be the best way we can add another contributor. From there, I can add to the harvester for obis specific data (api calls for required datasets and the processing needed for it). — Reply to this email directly, view it on GitHub <#11 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHICYOIISBQVORAHJZJCAWL2PX3TDAVCNFSM6AAAAABW3KMFUOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMRQGEZTQNY> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[JessyBarrette]

The metadata-entry-form project is more complex and relies on a google firebase project. We cannot run it locally via docker compose.

The can call above is only used by the explorer to retrieve more metadata from the different erddap datasets and assign a French title.

[vmehraOCN]

Yeah I was having issues getting firebase going locally without it connecting to our prod instances. I think it might be possible to emulate some of the functions, but its likely not needed at the moment.

My point with the above call is to leverage the current function and place it into an OBIS specific runner that would collect metadata and source information from CKAN. The prerequisite for this is that metadata entry form can accept OBIS as a source and populate CKAN with the entry. Once this is done I can setup another harvest ingestion stream with OBIS as the target asset and generate the data needed for CDE with api calls and transforming datasets as needed.

def list_ckan_records_with_OBIS_urls(cache_requests):
    while records_remaining:
        OBIS_datasets_query = (
            CKAN_API_URL
            + f"/action/package_search?rows={row_page_limit}&start={row_start}&q=obis"
        )

I found a sample of previous metadata that PSF submitted that has OBIS as the defined source, so I think this should be possible to integrate into just one module level of change.

[JessyBarrette]

We should probably move that discussion to the metadata repository. But yes, there's been work in the past to map DOI data cites to the CIOOS form and potentially OBIS. Not sure if a DOI is generated when a OBIS dataset is generated but if yes then we could perhaps just harvest the metadata from Datacite, if those are different then we would need a dedicated one for OBIS.

related to cioos-siooc/metadata-entry-form#386

[JessyBarrette]

But yes, this is two different topics. CDE harvester could certainly rely on the resources listed within the different CKAN datasets to harvest specific datasets from OBIS. That would be the ideal method I believe.

[vmehraOCN]

Sure, moving the discussion over there in the meantime. Once the metadata entry form is addressed with the ability to ingest the DOI from OBIS, its possible to continue the second half here for harvesting the data and adding it to cde.