Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/535 stratification diag #540

Draft
wants to merge 168 commits into
base: develop
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
168 commits
Select commit Hold shift + click to select a range
d424bbc
tidy variable names. Improve docstr
jpolton Sep 8, 2022
619192c
Apply Black formatting to Python code.
Sep 8, 2022
7d71fc7
rename class and camel-case methods. tidy variable names. Improve docstr
jpolton Sep 8, 2022
2ce1fe9
Apply Black formatting to Python code.
Sep 8, 2022
ac2986e
boolean test error
jpolton Sep 8, 2022
5d0f258
Merge branch 'develop' into feature/535_stratification_diag
jpolton Sep 9, 2022
d515672
Apply Black formatting to Python code.
Sep 9, 2022
0b9bf63
Merge branch 'develop' into feature/535_stratification_diag
jpolton Sep 9, 2022
675d5b6
Merge branch 'develop' into feature/535_stratification_diag
jpolton Sep 16, 2022
9cac654
InternalTide() --> GriddedStratification()
jpolton Sep 16, 2022
58a723d
Merge branch 'develop' into feature/535_stratification_diag
jpolton Sep 21, 2022
a7df4b3
Merge branch 'develop' into feature/535_stratification_diag
jpolton Nov 11, 2022
695b955
Merge branch 'develop' into feature/535_stratification_diag
jpolton Nov 12, 2022
c8e7dd6
Typo: S and T, not S and S
jpolton Nov 12, 2022
ecbebe1
new feature: fills_holes_1d()
jpolton Nov 12, 2022
2519e80
Apply Black formatting to Python code.
Nov 12, 2022
19d17f5
Commit generated unit test contents.
Nov 12, 2022
86d6cdd
Remove duplicate code
jpolton Nov 16, 2022
6cdc7a7
add profile.calculate_vertical_spacing()
jpolton Nov 17, 2022
97e901e
add profile.construct_density()
jpolton Nov 17, 2022
c0dcd7a
add profile.calculate_vertical_mask()
jpolton Nov 17, 2022
357fb1f
WIP: profile_stratification.py
jpolton Nov 17, 2022
eb37937
Add tests for profile.calculate_Vertical_spacing()
jpolton Nov 17, 2022
952a68d
Add tests for profile.calculate_vertical_mask()
jpolton Nov 17, 2022
764d243
Apply Black formatting to Python code.
Nov 17, 2022
b666d54
Commit generated unit test contents.
Nov 17, 2022
2a4ca40
calculate_vertical_mask(): made inputs same between Gridded and Profi…
jpolton Nov 18, 2022
6bb3724
add test for pPofile.contruct.density()
jpolton Nov 18, 2022
ea6f553
Apply Black formatting to Python code.
Nov 18, 2022
90f3ed4
Commit generated unit test contents.
Nov 18, 2022
2e4b67b
add ProfileStratification.calc_pea() and .quick_plot()
jpolton Nov 18, 2022
2e2e349
add test for GriddedStratification.calc_pea()
jpolton Nov 18, 2022
2218564
add test for ProfileStratification.calc_pea() and .quick_plot()
jpolton Nov 18, 2022
1976961
update unit_test framework with new, aligned, profile and gridded tests
jpolton Nov 18, 2022
78c5f88
update imports with new ProfileStratification class
jpolton Nov 18, 2022
8502d4f
update unit_test contents with new Gridded, GriddedStratification and…
jpolton Nov 18, 2022
e8d0fa6
Merge remote-tracking branch 'origin/feature/535_stratification_diag'…
jpolton Nov 18, 2022
7b542aa
fix nan_helper tests
jpolton Nov 18, 2022
4cfc6d4
fix test for calculate vertical mask
jpolton Nov 18, 2022
0331717
fix quick_plot
jpolton Nov 18, 2022
655029d
missing comma
jpolton Nov 18, 2022
af1c377
Apply Black formatting to Python code.
Nov 18, 2022
3e37876
Commit generated unit test contents.
Nov 18, 2022
2bac177
improve docstr
jpolton Nov 18, 2022
388be05
calculate_vertical_spacing(). Remove redundant if-not xarray condition
jpolton Nov 21, 2022
fb653c8
remove debugging print statements
jpolton Nov 21, 2022
e382311
Apply Black formatting to Python code.
Nov 21, 2022
101a9f5
Add profile PEA notebook
jpolton Nov 21, 2022
c7cab7b
Merge remote-tracking branch 'origin/feature/535_stratification_diag'…
jpolton Nov 21, 2022
6c91b75
Add clean profile PEA notebook
jpolton Nov 21, 2022
e6395bf
Merge branch 'develop' into feature/535_stratification_diag
jpolton Nov 23, 2022
467a172
profile.py Zd_max changed to use w-levels, order of variables in cons…
Nov 25, 2022
1e8123d
Apply Black formatting to Python code.
Nov 25, 2022
3c0894c
Merge branch 'develop' into feature/535_stratification_diag
jpolton Nov 28, 2022
b45fa6e
Adding match to grid method
jasontempestholt Nov 30, 2022
09673df
Adding match to grid method
jasontempestholt Nov 30, 2022
75bf680
Apply Black formatting to Python code.
Nov 30, 2022
6893ca1
Added gridded_to_profile_2d method to profile.py
jasontempestholt Dec 2, 2022
7ef0030
Apply Black formatting to Python code.
Dec 2, 2022
5214fa9
Added added unit test for gridded_to_profile_2d method to profile.py
jasontempestholt Dec 2, 2022
edffba2
Merge branch 'feature/535_stratification_diag' of https://github.com/…
jasontempestholt Dec 2, 2022
e27c3c6
Apply Black formatting to Python code.
Dec 2, 2022
6a83f7b
Find good SST and SSS depths in profiles
jasontempestholt Dec 6, 2022
e1a0a08
Merge branch 'feature/535_stratification_diag' of https://github.com/…
jasontempestholt Dec 7, 2022
fd33e83
Apply Black formatting to Python code.
Dec 7, 2022
7db197e
Profile analysis
jasontempestholt Dec 8, 2022
2f8521c
Apply Black formatting to Python code.
Dec 8, 2022
5721b19
Update profile_stratification.py
jasontempestholt Dec 15, 2022
b188dc8
Added option to subset dataset and domain on loading. This reduces th…
Dec 20, 2022
e591188
Apply Black formatting to Python code.
Dec 20, 2022
8372834
Merge branch 'develop' into feature/535_stratification_diag
jpolton Jan 4, 2023
e63262c
Merge branch 'feature/535_stratification_diag_tmp' into feature/535_s…
jpolton Jan 4, 2023
ecf9075
update gridded_monthly_hydrographic_climatology.py
Feb 8, 2023
19534c4
Apply Black formatting to Python code.
Feb 8, 2023
71e5f11
(old?) updates to profile_stratification.py
Jul 19, 2023
6463a6b
Update potential_energy_tutorial.ipynb
jasontempestholt Aug 30, 2023
ebe6951
Bug fixes to cleaning data - need to check this works ok. Update to n…
jasontempestholt Oct 20, 2023
eba39b9
merge develop
soutobias Nov 15, 2023
2d39c94
Apply Black formatting to Python code.
Sep 8, 2022
d72cf73
correct init
soutobias Nov 15, 2023
ed58839
Apply Black formatting to Python code.
Sep 8, 2022
fde09e0
boolean test error
jpolton Sep 8, 2022
0701201
InternalTide() --> GriddedStratification()
jpolton Sep 16, 2022
cc9d43e
Typo: S and T, not S and S
jpolton Nov 12, 2022
b20068c
new feature: fills_holes_1d()
jpolton Nov 12, 2022
f0dfda4
Apply Black formatting to Python code.
Nov 12, 2022
c1c124c
Commit generated unit test contents.
Nov 12, 2022
1804338
add profile.calculate_vertical_spacing()
jpolton Nov 17, 2022
7b0b085
add profile.construct_density()
jpolton Nov 17, 2022
8f33b37
add profile.calculate_vertical_mask()
jpolton Nov 17, 2022
5ca23f8
WIP: profile_stratification.py
jpolton Nov 17, 2022
ccf00e4
Add tests for profile.calculate_Vertical_spacing()
jpolton Nov 17, 2022
cb710ee
Add tests for profile.calculate_vertical_mask()
jpolton Nov 17, 2022
e69f797
Apply Black formatting to Python code.
Nov 17, 2022
b14e3d8
Commit generated unit test contents.
Nov 17, 2022
1887762
calculate_vertical_mask(): made inputs same between Gridded and Profi…
jpolton Nov 18, 2022
fa4adcb
add test for pPofile.contruct.density()
jpolton Nov 18, 2022
e7ef785
add ProfileStratification.calc_pea() and .quick_plot()
jpolton Nov 18, 2022
565dff8
correct conflict
soutobias Nov 15, 2023
47c3d61
add test for ProfileStratification.calc_pea() and .quick_plot()
jpolton Nov 18, 2022
2837669
update unit_test framework with new, aligned, profile and gridded tests
jpolton Nov 18, 2022
39e9240
correct conflict
soutobias Nov 15, 2023
bdcebc3
update unit_test contents with new Gridded, GriddedStratification and…
jpolton Nov 18, 2022
58b6a45
Apply Black formatting to Python code.
Nov 18, 2022
84e186e
Commit generated unit test contents.
Nov 18, 2022
83bba4e
fix nan_helper tests
jpolton Nov 18, 2022
a141a68
fix test for calculate vertical mask
jpolton Nov 18, 2022
932d4e4
fix quick_plot
jpolton Nov 18, 2022
de847e9
missing comma
jpolton Nov 18, 2022
7f01c24
Apply Black formatting to Python code.
Nov 18, 2022
1140dd2
correct conflict
soutobias Nov 15, 2023
b08f1a2
improve docstr
jpolton Nov 18, 2022
4035b9f
calculate_vertical_spacing(). Remove redundant if-not xarray condition
jpolton Nov 21, 2022
f5150fd
remove debugging print statements
jpolton Nov 21, 2022
17602f5
correct conflict
soutobias Nov 15, 2023
f98f679
Apply Black formatting to Python code.
Nov 21, 2022
e49ff72
Add clean profile PEA notebook
jpolton Nov 21, 2022
bde5537
profile.py Zd_max changed to use w-levels, order of variables in cons…
Nov 25, 2022
732a415
Apply Black formatting to Python code.
Nov 25, 2022
3425ef0
correct conflict
soutobias Nov 15, 2023
5a9c4fd
Adding match to grid method
jasontempestholt Nov 30, 2022
5f99bc9
correct conflict
soutobias Nov 15, 2023
e25d238
Added gridded_to_profile_2d method to profile.py
jasontempestholt Dec 2, 2022
1c09a98
Added added unit test for gridded_to_profile_2d method to profile.py
jasontempestholt Dec 2, 2022
1de170e
correct conflict
soutobias Nov 15, 2023
b787da5
Find good SST and SSS depths in profiles
jasontempestholt Dec 6, 2022
e63d4bb
correct conflict
soutobias Nov 15, 2023
73eab1c
Apply Black formatting to Python code.
Dec 7, 2022
7704a8e
Profile analysis
jasontempestholt Dec 8, 2022
d03f279
Apply Black formatting to Python code.
Dec 8, 2022
e5f951a
Update profile_stratification.py
jasontempestholt Dec 15, 2022
3e30fbe
correct conflict
soutobias Nov 15, 2023
cbdb6e5
correct conflict
soutobias Nov 15, 2023
78be971
update gridded_monthly_hydrographic_climatology.py
Feb 8, 2023
7e08f79
correct conflict
soutobias Nov 15, 2023
aeb1408
(old?) updates to profile_stratification.py
Jul 19, 2023
c41c59e
Update potential_energy_tutorial.ipynb
jasontempestholt Aug 30, 2023
a6574ef
Bug fixes to cleaning data - need to check this works ok. Update to n…
jasontempestholt Oct 20, 2023
2f81b38
add
soutobias Nov 15, 2023
06f3085
create notebook and part of the tests for monthly climatology and zarr
soutobias Nov 17, 2023
aab4977
correct conflits
soutobias Nov 17, 2023
b7e3686
Apply Black formatting to Python code.
Nov 17, 2023
736514a
Commit generated unit test contents.
Nov 17, 2023
8296722
Resorting to previous climateology behaviour
jasontempestholt Dec 19, 2023
08331ff
update strat analysis
Jan 16, 2024
1979b2d
Update with develop
jasontempestholt Jan 16, 2024
e4153bf
Apply Black formatting to Python code.
Jan 16, 2024
4d3d54c
Adding match-to-grid capability to profile_stratification.py and corr…
jasontempestholt Jan 19, 2024
e4d4ad7
Merge branch 'feature/535_stratification_diag' of github.com:British-…
jasontempestholt Jan 19, 2024
22f45d8
Apply Black formatting to Python code.
Jan 19, 2024
75b5fa4
lablel indices with grid name incase want to store indices from mult…
jasontempestholt Jan 19, 2024
6a075fb
Apply Black formatting to Python code.
Jan 19, 2024
d4af911
Update profile_stratification.py
jasontempestholt Jan 25, 2024
e3ada95
Apply Black formatting to Python code.
Jan 25, 2024
31ebee5
Update with develop
jasontempestholt Jan 26, 2024
8d1a275
Apply Black formatting to Python code.
Jan 26, 2024
ccddc03
Updates to profile and stratification
jasontempestholt Jan 29, 2024
dab27e0
Apply Black formatting to Python code.
Jan 29, 2024
60fafc4
Update profile.py
jasontempestholt Jan 29, 2024
828f195
Merge branch 'feature/535_stratification_diag' of https://github.com/…
jasontempestholt Jan 29, 2024
70879df
Apply Black formatting to Python code.
Jan 29, 2024
634225f
permit TEOS10 conserv temp and abs sal
jpolton May 13, 2024
b9e9d15
Apply Black formatting to Python code.
May 13, 2024
21782c4
Update profile_stratification.py
jpolton May 14, 2024
15850ee
Apply Black formatting to Python code.
May 14, 2024
0d46d1a
Update profile_stratification.py
jpolton May 14, 2024
5227549
Update profile_stratification.py
jpolton May 14, 2024
a3e716a
Apply Black formatting to Python code.
May 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Updates to profile and stratification
Passing variables needed for match to grid
Remove this version of match to grid as provided in profile

Profile sub setting across latitude wrap
jasontempestholt committed Jan 29, 2024
commit ccddc03ee55d8abd00daae14be28a25e79837a98
35 changes: 27 additions & 8 deletions coast/data/profile.py
Original file line number Diff line number Diff line change
@@ -153,9 +153,19 @@ def subset_indices_lonlat_box(self, lonbounds, latbounds):

return: A new profile object containing subsetted data
"""
ind = general_utils.subset_indices_lonlat_box(
self.dataset.longitude, self.dataset.latitude, lonbounds[0], lonbounds[1], latbounds[0], latbounds[1]
if lonbounds[0] < lonbounds[1]:
ind = general_utils.subset_indices_lonlat_box(
self.dataset.longitude, self.dataset.latitude, lonbounds[0], lonbounds[1], latbounds[0], latbounds[1]
)
else:
ind1 = general_utils.subset_indices_lonlat_box(
self.dataset.longitude, self.dataset.latitude, lonbounds[0], 180.0 , latbounds[0], latbounds[1]
)
ind2 = general_utils.subset_indices_lonlat_box(
self.dataset.longitude, self.dataset.latitude, -180.0, lonbounds[1], latbounds[0], latbounds[1]
)
ind={}
ind[0] = np.concatenate((ind1[0],ind2[0]))
return Profile(dataset=self.dataset.isel(id_dim=ind[0]))

def extract_en4_profiles(self, dataset_names, region_bounds, chunks: dict = {}):
@@ -173,6 +183,7 @@ def extract_en4_profiles(self, dataset_names, region_bounds, chunks: dict = {}):
y_max = region_bounds[3]
# self.profile = Profile(config=config)
self.read_en4(dataset_names, multiple=True, chunks=chunks)

pr = self.subset_indices_lonlat_box(lonbounds=[x_min, x_max], latbounds=[y_min, y_max])
pr = pr.process_en4()
return pr
@@ -498,7 +509,7 @@ def obs_operator(self, gridded, mask_bottom_level=True):
mod_profiles["nearest_index_t"] = (["id_dim"], ind_t.values)
return Profile(dataset=mod_profiles)

def match_to_grid(self, gridded, limits=[0, 0, 0, 0], rmax=7.0) -> None:
def match_to_grid(self, gridded, limits=[0, 0, 0, 0], rmax=25.0) -> None:
"""Match profiles locations to grid, finding 4 nearest neighbours for each profile.

Args:
@@ -516,13 +527,17 @@ def match_to_grid(self, gridded, limits=[0, 0, 0, 0], rmax=7.0) -> None:
"""

if sum(limits) != 0:
gridded.subset(ydim=range(limits[0], limits[1] + 0), xdim=range(limits[2], limits[3] + 1))
gridded.subset(y_dim=range(limits[0], limits[1] + 1), x_dim=range(limits[2], limits[3] + 1))
# keep the grid or subset on the hydrographic profiles object
gridded.dataset["limits"] = limits

prf = self.dataset
grd = gridded.dataset
grd["landmask"] = grd.bottom_level == 0
if "bottom_level" in grd:
grd["landmask"] = grd.bottom_level == 0
else: #resort to using bathymetry
grd["landmask"] = grd.bathymetry == 0

lon_prf = prf["longitude"]
lat_prf = prf["latitude"]
lon_grd = grd["longitude"]
@@ -590,7 +605,7 @@ def match_to_grid(self, gridded, limits=[0, 0, 0, 0], rmax=7.0) -> None:
self.dataset["rmin_prf"] = xr.DataArray(rmin_prf, dims=["id_dim", "NNs"])
self.dataset["ind_good"] = xr.DataArray(ind_good, dims=["Ngood"])

def gridded_to_profile_2d(self, gridded, variable) -> None:
def gridded_to_profile_2d(self, gridded, variable,limits=[0,0,0,0],rmax=25.0) -> None:
"""
Evaluated a gridded data variable on each profile. Here just 2D, but could be extended to 3 or 4D

@@ -605,11 +620,15 @@ def gridded_to_profile_2d(self, gridded, variable) -> None:
"""
# ensure there are indices in profile
if not "ind_x" in self.dataset:
self.match_to_grid(gridded)
self.match_to_grid(gridded,limits=limits,rmax=rmax)
#
prf = self.dataset
grd = gridded.dataset
grd["landmask"] = grd.bottom_level == 0
if "botton_level" in grd:
grd["landmask"] = grd.bottom_level == 0
else: # resort to bathymetry for mask
grd["landmask"] = grd.bathymetry == 0

nprof = self.dataset.id_dim.shape[0]
var = np.ma.masked_where(grd["landmask"], grd[variable])
ig = prf.ind_good
75 changes: 3 additions & 72 deletions coast/diagnostics/profile_stratification.py
Original file line number Diff line number Diff line change
@@ -39,7 +39,7 @@ def __init__(self, profile: xr.Dataset):
self.nz = profile.dataset.dims["z_dim"]
debug(f"Initialised {get_slug(self)}")

def clean_data(profile: xr.Dataset, gridded: xr.Dataset, Zmax):
def clean_data(profile: xr.Dataset, gridded: xr.Dataset, Zmax,limits=[0,0,0,0],rmax=25.):
"""
Cleaning data for stratification metric calculations
Stage 1:...
@@ -68,7 +68,7 @@ def first_nonzero(arr, axis=0, invalid_val=np.nan):
return np.where(mask.any(axis=axis), mask.argmax(axis=axis), invalid_val)

if "bathymetry" in gridded.dataset:
profile.gridded_to_profile_2d(gridded, "bathymetry")
profile.gridded_to_profile_2d(gridded, "bathymetry",limits=limits,rmax=rmax)
D_prf = profile.dataset.bathymetry.values
z = profile.dataset.depth
test_surface = z < np.minimum(dz_max, 0.25 * np.repeat(D_prf[:, np.newaxis], n_depth, axis=1))
@@ -143,7 +143,7 @@ def first_nonzero(arr, axis=0, invalid_val=np.nan):
# %%
return profile

def calc_pea(self, profile: xr.Dataset, gridded: xr.Dataset, Zmax):
def calc_pea(self, profile: xr.Dataset, gridded: xr.Dataset, Zmax, rmax=25.0, limits=[0,0,0,0]):
"""
Calculates Potential Energy Anomaly

@@ -250,72 +250,3 @@ def quick_plot(self, var: xr.DataArray = None):
return fig, ax

##############################################################################
def match_to_grid(self, gridded: xr.Dataset, limits: List = [0, 0, 0, 0], rmax=7000.0, grid_name="prf") -> None:
"""Match profiles locations to grid, finding 4 nearest neighbours for each profile.

Args:
gridded (Gridded): Gridded object.
limits (List): [jmin,jmax,imin,imax] - Subset to this region.
rmax (int): 7000 m - maxmimum search distance (metres).

### NEED TO DESCRIBE THE OUTPUT. WHAT DO i_prf, j_prf, rmin_prf REPRESENT?

### THIS LOOKS LIKE SOMETHING THE profile.obs_operator WOULD DO
"""

if sum(limits) != 0:
gridded.subset(y_dim=range(limits[0], limits[1] + 1), x_dim=range(limits[2], limits[3] + 1))
# gridded.spatial _subset(limits) might need this one for wrapping
# keep the bathymetry and mask or subset on the hydrographic profiles object
gridded.dataset["limits"] = limits

lon_prf = self.dataset.longitude.values
lat_prf = self.dataset.latitude.values

# Find 4 nearest neighbours on grid
j_prf, i_prf, rmin_prf = gridded.find_j_i_list(lat=lat_prf, lon=lon_prf, n_nn=4)

self.dataset["i_min"] = limits[0] # reference back to origianl grid
self.dataset["j_min"] = limits[2]

i_min = self.dataset.i_min.values
j_min = self.dataset.j_min.values

# Sort 4 NN by distance on grid
ii = np.nonzero(np.isnan(lon_prf))
i_prf[ii, :] = 0
j_prf[ii, :] = 0
ip = np.where(np.logical_or(i_prf[:, 0] != 0, j_prf[:, 0] != 0))[0]
lon_prf4 = np.repeat(lon_prf[ip, np.newaxis], 4, axis=1).ravel()
lat_prf4 = np.repeat(lat_prf[ip, np.newaxis], 4, axis=1).ravel()
r = np.ones(i_prf.shape) * np.nan
lon_grd = gridded.dataset.longitude.values
lat_grd = gridded.dataset.latitude.values

rr = ProfileStratification.distance_on_grid(
lat_grd, lon_grd, j_prf[ip, :].ravel(), i_prf[ip, :].ravel(), lat_prf4, lon_prf4
)
r[ip, :] = np.reshape(rr, (ip.size, 4))
# sort by distance
ii = np.argsort(r, axis=1)
rmin_prf = np.take_along_axis(r, ii, axis=1)
i_prf = np.take_along_axis(i_prf, ii, axis=1)
j_prf = np.take_along_axis(j_prf, ii, axis=1)

ii = np.nonzero(np.logical_or(np.min(r, axis=1) > rmax, np.isnan(lon_prf)))
i_prf = i_prf + i_min
j_prf = j_prf + j_min
i_prf[ii, :] = 0 # should the be nan?
j_prf[ii, :] = 0

self.dataset[f"i_{grid_name}"] = xr.DataArray(i_prf, dims=["id_dim", "4"])
self.dataset[f"j_{grid_name}"] = xr.DataArray(j_prf, dims=["id_dim", "4"])
self.dataset[f"rmin_{grid_name}"] = xr.DataArray(rmin_prf, dims=["id_dim", "4"])
# self.dataset[f"bathy_{grid_name}"] = gridded.dataset.bathymetry
# self.dataset[f"mask_{grid_name}"] = gridded.dataset.bottom_level != 0

def distance_on_grid(Y, X, jpts, ipts, Ypts, Xpts):
DX = (Xpts - X[jpts, ipts]) * earth_radius * np.cos(Ypts * np.pi / 180.0)
DY = (Ypts - Y[jpts, ipts]) * earth_radius
r = np.sqrt(DX**2 + DY**2)
return r