Updating native 4D-Var to IODA observation files converter

[hga007]

Description

This PR updates the native ROMS to the IODA-type observation files converter, roms2ioda.m. The generic JEDI data assimilation framework uses the IODA-type observation files. It is an enhanced NetCDF-4 with first-level grouping and the extension nc4 for differentiation.

Usually, one IODA-type file is available for each variable in the control vector, like SSH, SST, SSS, HF radar velocities, temperature, and salinity files for a particular data assimilation cycle. A file can include multiple variables representing vectors (u- and v-momentum) and spectral data with exact coordinates (lon, lat, depth, time). It may also contain multiple scalar variables if at the precise location, such as T and S profiles or biological profiles for various ecosystem state variables. I may also include secondary variables to compute a control variable with an elaborate operator. This setup gives us a lot of flexibility for filtering and averaging operators.

The native ROMS 4D-Var data assimilation operators are being restructured to allow multiple IODA-type files. This facilitates area-averaged and/or time-averaged H(x) operators that compute model solutions at the observation locations. This capability will be available soon.

---

The main objective of the restructuring is to have IODA-type observation files that can be used either in the native ROMS 4D-Var drivers or in any of the ROMS-JEDI data assimilation algorithms. Also, sophisticated H(x) operators will be available in the native ROMS 4D-Var algorithms. For more details, please check myroms/roms#62.

---

For example, the IODA-type NetCDF-4 schema for SSH observations that require an area-averaged (60 km) and time-averaged (36-hour) filter in the H(x) operator is as follows:

% ncdump -t -v dateTimeAverageBegin,dateTimeAverageEnd,spatialAverage usec3_adt_20190827.nc4

netcdf usec3_adt_20190827 {
dimensions:
	Location = 1940 ;
	nvars = 1 ;
	survey = 17 ;
	timeWindow = 2 ;
variables:
	int Location(Location) ;
		Location:suggested_chunck_dim = 512 ;
	int nvars(nvars) ;
		nvars:suggested_chunck_dim = 100 ;
	int survey(survey) ;
		survey:suggested_chunck_dim = 100 ;
	int timeWindow(timeWindow) ;
		timeWindow:suggested_chunck_dim = 100 ;

// global attributes:
		:_ioda_layout = "ObsGroup" ;
		:_ioda_layout_version = 3 ;
		:odb_version = 1 ;
		:date_time = 2019082700. ;
		:datetimeReference = "2019-08-27T00:00:00Z" ;
		:description = "Native ROMS 4D-Var observations file converted to IODA" ;
		:sourceFiles = "usec3km_roms_obs_20190827.nc" ;
		:history = "Created from Matlab script: create_ioda_obs on Monday - August 4, 2025 - 3:25:57.5804 PM" ;
data:

group: MetaData {
  variables:
  	int64 dateTime(Location) ;
  		dateTime:long_name = "elapsed observation time since reference" ;
  		dateTime:units = "seconds since 2019-08-27T00:00:00Z" ;
  	int64 dateTimeAverageBegin(timeWindow) ;
  		dateTimeAverageBegin:long_name = "start of time averaging filter" ;
  		dateTimeAverageBegin:units = "seconds since 2019-08-27T00:00:00Z" ;
  	int64 dateTimeAverageEnd(timeWindow) ;
  		dateTimeAverageEnd:long_name = "end of time averaging filter" ;
  		dateTimeAverageEnd:units = "seconds since 2019-08-27T00:00:00Z" ;
  	float latitude(Location) ;
  		latitude:long_name = "observation latitude" ;
  		latitude:units = "degrees_north" ;
  	float longitude(Location) ;
  		longitude:long_name = "observation longitude" ;
  		longitude:units = "degrees_east" ;
  	int provenance(Location) ;
  		provenance:long_name = "observation origin identifier" ;
  		provenance:negative = "repetitive observation at different times and error" ;
  	int sequenceNumber(Location) ;
  		sequenceNumber:long_name = "observation sequence number" ;
  	float spatialAverage ;
  		spatialAverage:long_name = "half-length of spatial averaging filter" ;
  		spatialAverage:units = "meter" ;
  	int stateID(nvars) ;
  		stateID:long_name = "state variable index" ;
  	int surveyIndex(survey) ;
  		surveyIndex:long_name = "observation survey time indices as they appear in dateTime" ;
  	int64 surveyTime(survey) ;
  		surveyTime:long_name = "observation survey time" ;
  		surveyTime:units = "seconds since 2019-08-27T00:00:00Z" ;
  	string variables_name(nvars) ;
  		variables_name:long_name = "observation UFO/IODA standard name" ;
  		string variables_name:_FillValue = "" ;
  	float x_grid(Location) ;
  		x_grid:long_name = "observation fractional x-grid location" ;
  	float y_grid(Location) ;
  		y_grid:long_name = "observation fractional y-grid location" ;
  data:

   dateTimeAverageBegin = "2019-08-27", "2019-08-28 12" ;

   dateTimeAverageEnd = "2019-08-28 12", "2019-08-30" ;

   spatialAverage = 30000 ;
  } // group MetaData

group: ObsError {
  variables:
  	float absoluteDynamicTopography(Location) ;
  		absoluteDynamicTopography:long_name = "observation error standard deviation" ;
  		absoluteDynamicTopography:units = "meter" ;
  		absoluteDynamicTopography:coordinates = "longitude, latitude, dateTime" ;
  data:
  } // group ObsError

group: ObsValue {
  variables:
  	float absoluteDynamicTopography(Location) ;
  		absoluteDynamicTopography:long_name = "observation value" ;
  		absoluteDynamicTopography:units = "meter" ;
  		absoluteDynamicTopography:coordinates = "longitude, latitude, dateTime" ;
  data:
  } // group ObsValue

group: PreQC {
  variables:
  	int absoluteDynamicTopography(Location) ;
  		absoluteDynamicTopography:long_name = "observation preset quality control filter identifier" ;
  		absoluteDynamicTopography:coordinates = "longitude, latitude, dateTime" ;
  data:
  } // group PreQC
}

---

For HF radar velocity observations at a depth of 2 m, which computes an H(x) with a 24-hour time-averaged filter :

% ncdump -t -v dateTimeAverageBegin,dateTimeAverageEnd usec3_uv_codar_20190827.nc4

netcdf usec3_uv_codar_20190827 {
dimensions:
	Location = 10332 ;
	nvars = 2 ;
	survey = 72 ;
	timeWindow = 3 ;
variables:
	int Location(Location) ;
		Location:suggested_chunck_dim = 512 ;
	int nvars(nvars) ;
		nvars:suggested_chunck_dim = 100 ;
	int survey(survey) ;
		survey:suggested_chunck_dim = 100 ;
	int timeWindow(timeWindow) ;
		timeWindow:suggested_chunck_dim = 100 ;

// global attributes:
		:_ioda_layout = "ObsGroup" ;
		:_ioda_layout_version = 3 ;
		:odb_version = 1 ;
		:date_time = 2019082700. ;
		:datetimeReference = "2019-08-27T00:00:00Z" ;
		:description = "Native ROMS 4D-Var observations file converted to IODA" ;
		:sourceFiles = "usec3km_roms_obs_20190827.nc" ;
		:history = "Created from Matlab script: create_ioda_obs on Monday - August 4, 2025 - 3:26:0.62598 PM" ;
data:

group: MetaData {
  variables:
  	int64 dateTime(Location) ;
  		dateTime:long_name = "elapsed observation time since reference" ;
  		dateTime:units = "seconds since 2019-08-27T00:00:00Z" ;
  	int64 dateTimeAverageBegin(timeWindow) ;
  		dateTimeAverageBegin:long_name = "start of time averaging filter" ;
  		dateTimeAverageBegin:units = "seconds since 2019-08-27T00:00:00Z" ;
  	int64 dateTimeAverageEnd(timeWindow) ;
  		dateTimeAverageEnd:long_name = "end of time averaging filter" ;
  		dateTimeAverageEnd:units = "seconds since 2019-08-27T00:00:00Z" ;
  	float depth(Location) ;
  		depth:long_name = "observation depth below sea level" ;
  		depth:units = "meter" ;
  		depth:negative = "downwards" ;
  	float latitude(Location) ;
  		latitude:long_name = "observation latitude" ;
  		latitude:units = "degrees_north" ;
  	float longitude(Location) ;
  		longitude:long_name = "observation longitude" ;
  		longitude:units = "degrees_east" ;
  	int provenance(Location) ;
  		provenance:long_name = "observation origin identifier" ;
  	int sequenceNumber(Location) ;
  		sequenceNumber:long_name = "observation sequence number" ;
  	int stateID(nvars) ;
  		stateID:long_name = "state variable index" ;
  	int surveyIndex(survey) ;
  		surveyIndex:long_name = "observation survey time indices as they appear in dateTime" ;
  	int64 surveyTime(survey) ;
  		surveyTime:long_name = "observation survey time" ;
  		surveyTime:units = "seconds since 2019-08-27T00:00:00Z" ;
  	string variables_name(nvars) ;
  		variables_name:long_name = "observation UFO/IODA standard name" ;
  		string variables_name:_FillValue = "" ;
  	float x_grid(Location) ;
  		x_grid:long_name = "observation fractional x-grid location" ;
  	float y_grid(Location) ;
  		y_grid:long_name = "observation fractional y-grid location" ;
  	float z_grid(Location) ;
  		z_grid:long_name = "observation fractional z-grid location" ;
  data:

   dateTimeAverageBegin = "2019-08-27", "2019-08-28", "2019-08-29" ;

   dateTimeAverageEnd = "2019-08-28", "2019-08-29", "2019-08-30" ;
  } // group MetaData

group: ObsError {
  variables:
  	float waterZonalVelocity(Location) ;
  		waterZonalVelocity:long_name = "observation error standard deviation" ;
  		waterZonalVelocity:units = "m s-1" ;
  		waterZonalVelocity:coordinates = "longitude, latitude, depth, dateTime" ;
  	float waterMeridionalVelocity(Location) ;
  		waterMeridionalVelocity:long_name = "observation error standard deviation" ;
  		waterMeridionalVelocity:units = "m s-1" ;
  		waterMeridionalVelocity:coordinates = "longitude, latitude, depth, dateTime" ;
  data:
  } // group ObsError

group: ObsValue {
  variables:
  	float waterZonalVelocity(Location) ;
  		waterZonalVelocity:long_name = "observation value" ;
  		waterZonalVelocity:units = "m s-1" ;
  		waterZonalVelocity:coordinates = "longitude, latitude, depth, dateTime" ;
  	float waterMeridionalVelocity(Location) ;
  		waterMeridionalVelocity:long_name = "observation value" ;
  		waterMeridionalVelocity:units = "m s-1" ;
  		waterMeridionalVelocity:coordinates = "longitude, latitude, depth, dateTime" ;
  data:
  } // group ObsValue

group: PreQC {
  variables:
  	int waterZonalVelocity(Location) ;
  		waterZonalVelocity:long_name = "observation preset quality control filter identifier" ;
  		waterZonalVelocity:coordinates = "longitude, latitude, depth, dateTime" ;
  	int waterMeridionalVelocity(Location) ;
  		waterMeridionalVelocity:long_name = "observation preset quality control filter identifier" ;
  		waterMeridionalVelocity:coordinates = "longitude, latitude, depth, dateTime" ;
  data:
  } // group PreQC
}

---

These IODA-type observation files are easily generated with the roms2ioda.m conversion script:

>> Hname='/home/arango/ROMS/Projects/USEC/RBL4DVAR_mixres/Set01/2019.08.27usec3km_roms_fwd_20190827_outer0.nc';
>> cd /home/arango/ocean/repository/git/roms_test/USEC/Data/OBS

>> roms2ioda('usec3km_roms_obs_20190827.nc',Hname,'usec3','20190827',30,36,24);
 
*** Creating observations file:  usec3_adt_20190827.nc4
*** Writing  observations file:  usec3_adt_20190827.nc4
 
*** Creating observations file:  usec3_sst_20190827.nc4
*** Writing  observations file:  usec3_sst_20190827.nc4
 
*** Creating observations file:  usec3_temp_20190827.nc4
*** Writing  observations file:  usec3_temp_20190827.nc4
 
*** Creating observations file:  usec3_ptemp_20190827.nc4
*** Writing  observations file:  usec3_ptemp_20190827.nc4
 
*** Creating observations file:  usec3_salt_20190827.nc4
*** Writing  observations file:  usec3_salt_20190827.nc4
 
*** Creating observations file:  usec3_uv_codar_20190827.nc4
*** Writing  observations file:  usec3_uv_codar_20190827.nc4

Note

The native ROMS 4D-Var file usec3km_roms_obs_20190827.nc is converted to six IODA-type NetCDF-4 files.