Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: actris-cloudnet/cloudnetpy
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.68.1
Choose a base ref
...
head repository: actris-cloudnet/cloudnetpy
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref

Commits on Jan 29, 2025

  1. Copy the full SHA
    6689d1b View commit details
  2. Copy the full SHA
    24ffdc6 View commit details
  3. Release version 1.69.0

    siiptuo committed Jan 29, 2025
    Copy the full SHA
    c2a765b View commit details
  4. Copy the full SHA
    b63202c View commit details
  5. Release version 1.69.1

    siiptuo committed Jan 29, 2025
    Copy the full SHA
    24c21c1 View commit details
  6. Copy the full SHA
    3cc8a80 View commit details
  7. Release version 1.69.2

    siiptuo committed Jan 29, 2025
    Copy the full SHA
    76c2b0a View commit details

Commits on Jan 31, 2025

  1. Improve chm15k version check

    siiptuo committed Jan 31, 2025
    Copy the full SHA
    3777c27 View commit details
  2. Release version 1.69.3

    siiptuo committed Jan 31, 2025
    Copy the full SHA
    cbee33c View commit details
  3. Fix docs workflow

    siiptuo committed Jan 31, 2025
    Copy the full SHA
    57a08e7 View commit details

Commits on Feb 3, 2025

  1. Copy the full SHA
    20023bb View commit details
  2. Copy the full SHA
    abfb434 View commit details
  3. Save snr_limit as float

    tukiains committed Feb 3, 2025
    Copy the full SHA
    44261cf View commit details
  4. Release version 1.69.4

    tukiains committed Feb 3, 2025
    Copy the full SHA
    5372945 View commit details

Commits on Feb 4, 2025

  1. Copy the full SHA
    fa79655 View commit details
  2. Release version 1.69.5

    tukiains committed Feb 4, 2025
    Copy the full SHA
    5e901ee View commit details

Commits on Feb 5, 2025

  1. Copy the full SHA
    fb03fc0 View commit details
  2. Release version 1.69.6

    tukiains committed Feb 5, 2025
    Copy the full SHA
    dacc26d View commit details
  3. Test with Python 3.13

    siiptuo committed Feb 5, 2025
    Copy the full SHA
    b6ce03a View commit details
  4. Copy the full SHA
    5526bd6 View commit details

Commits on Feb 7, 2025

  1. Copy the full SHA
    06c0ff9 View commit details
  2. Release version 1.69.7

    siiptuo committed Feb 7, 2025
    Copy the full SHA
    2541051 View commit details

Commits on Feb 10, 2025

  1. Copy the full SHA
    1bc7a87 View commit details
  2. Release version 1.69.8

    siiptuo committed Feb 10, 2025
    Copy the full SHA
    36f24bd View commit details
  3. Copy the full SHA
    a573a6d View commit details
  4. Copy the full SHA
    8befa71 View commit details
  5. Release version 1.69.9

    siiptuo committed Feb 10, 2025
    Copy the full SHA
    25fd551 View commit details
  6. Copy the full SHA
    291537e View commit details
  7. Release version 1.69.10

    siiptuo committed Feb 10, 2025
    Copy the full SHA
    2c38b04 View commit details

Commits on Feb 11, 2025

  1. Add cpr-simulation plots

    tukiains committed Feb 11, 2025
    Copy the full SHA
    df72da7 View commit details
  2. Release version 1.70.0

    tukiains committed Feb 11, 2025
    Copy the full SHA
    f5af139 View commit details

Commits on Feb 13, 2025

  1. Copy the full SHA
    92bb6b4 View commit details
  2. Release version 1.70.1

    tukiains committed Feb 13, 2025
    Copy the full SHA
    20868eb View commit details
4 changes: 2 additions & 2 deletions .github/workflows/docs.yaml
Original file line number Diff line number Diff line change
@@ -26,7 +26,7 @@ jobs:
make html
touch html/.nojekyll
- name: Upload Pages artifact
uses: actions/upload-pages-artifact@v2
uses: actions/upload-pages-artifact@v3
with:
path: docs/html
deploy:
@@ -41,4 +41,4 @@ jobs:
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@v2
uses: actions/deploy-pages@v4
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -7,7 +7,7 @@ jobs:
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
python-version: ["3.10", "3.11", "3.12"]
python-version: ["3.10", "3.11", "3.12", "3.13"]

runs-on: ${{ matrix.os }}
steps:
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
input/
output/
*.nc
.idea/
.vscode/
venv/
56 changes: 56 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,61 @@
# Changelog

## 1.70.1 – 2025-02-13

- Add plotting metadata for EarthCARE velocity variables

## 1.70.0 – 2025-02-11

- Add cpr-simulation plotting

## 1.69.10 – 2025-02-10

- Screen invalid rainfall rates in Thies LPM

## 1.69.9 – 2025-02-10

- Skip null characters Lampedusa weather station
- Remove `WeatherStationDataError`

## 1.69.8 – 2025-02-10

- Add support for Lampedusa weather station

## 1.69.7 – 2025-02-07

- Support `time_offset` in `hatpro2l1c`
- Test with Python 3.13

## 1.69.6 – 2025-02-05

- Ignore masked mira timestamps

## 1.69.5 – 2025-02-04

- Fix bug in calibration data reading

## 1.69.4 – 2025-02-03

- Add snr_limit metadata option to mira
- Store snr_limit variable in netCDF

## 1.69.3 – 2025-01-31

- Improve `chm15k` version check

## 1.69.2 – 2025-01-29

- Handle invalid timestamps in CT25K

## 1.69.1 – 2025-01-29

- Support zenith and azimuth offsets in MIRA

## 1.69.0 – 2025-01-29

- Support Jülich weather station
- Add ARPEGE to supported models

## 1.68.1 – 2025-01-17

- Improve rain[e]H3 timestamp screening
4 changes: 2 additions & 2 deletions cloudnetpy/categorize/model.py
Original file line number Diff line number Diff line change
@@ -164,11 +164,11 @@ def _calc_mean_height(model_heights: np.ndarray) -> np.ndarray:

def _find_model_type(file_name: str) -> str:
"""Finds model type from the model filename."""
possible_keys = ("gdas1", "icon", "ecmwf", "harmonie", "era5")
possible_keys = ("gdas1", "icon", "ecmwf", "harmonie", "era5", "arpege")
for key in possible_keys:
if key in file_name:
return key
msg = "Unknown model type"
msg = f"Unknown model type: {file_name}"
raise ValueError(msg)


3 changes: 2 additions & 1 deletion cloudnetpy/constants.py
Original file line number Diff line number Diff line change
@@ -24,13 +24,14 @@
SEC_IN_DAY: Final = 86400
MM_TO_M: Final = 1e-3
G_TO_KG: Final = 1e-3
M_TO_KM: Final = 1e-3
KG_TO_G: Final = 1e3
M_TO_MM: Final = 1e3
M_S_TO_MM_H: Final = SEC_IN_HOUR / MM_TO_M
MM_H_TO_M_S: Final = 1 / M_S_TO_MM_H
GHZ_TO_HZ: Final = 1e9
HPA_TO_PA: Final = 100
PA_TO_HPA: Final = 1 / HPA_TO_PA
KM_H_TO_M_S: Final = 1000 / SEC_IN_HOUR
M_TO_KM: Final = 1e-3
TWO_WAY: Final = 2
G: Final = 9.80665
7 changes: 0 additions & 7 deletions cloudnetpy/exceptions.py
Original file line number Diff line number Diff line change
@@ -30,13 +30,6 @@ def __init__(self, msg: str):
super().__init__(msg)


class WeatherStationDataError(CloudnetException):
"""Internal exception class."""

def __init__(self, msg: str = "Unable to read the file"):
super().__init__(msg)


class ModelDataError(CloudnetException):
"""Internal exception class."""

2 changes: 1 addition & 1 deletion cloudnetpy/instruments/copernicus.py
Original file line number Diff line number Diff line change
@@ -97,7 +97,7 @@ def copernicus2nc(
copernicus.add_nyquist_velocity(keymap)
copernicus.add_site_geolocation()
valid_indices = copernicus.add_zenith_and_azimuth_angles(
elevation_threshold=1,
elevation_threshold=1.1,
elevation_diff_threshold=0.1,
azimuth_diff_threshold=0.1,
)
8 changes: 8 additions & 0 deletions cloudnetpy/instruments/disdrometer/thies.py
Original file line number Diff line number Diff line change
@@ -4,6 +4,7 @@
from typing import Any

import numpy as np
from numpy import ma

from cloudnetpy import output
from cloudnetpy.cloudnetarray import CloudnetArray
@@ -107,6 +108,7 @@ def thies2nc(
raise DisdrometerDataError(msg) from err
disdrometer.sort_timestamps()
disdrometer.remove_duplicate_timestamps()
disdrometer.mask_invalid_values()
disdrometer.add_meta()
disdrometer.convert_units()
attributes = output.add_time_attribute(ATTRIBUTES, disdrometer.date)
@@ -269,6 +271,12 @@ def _screen_time(self, expected_date: datetime.date | None = None) -> None:
for key in self.raw_data:
self.raw_data[key] = self.raw_data[key][valid_mask]

def mask_invalid_values(self) -> None:
rainfall_rate = self.data["rainfall_rate"]
rainfall_rate.data = ma.masked_where(
rainfall_rate.data > 999, rainfall_rate.data
)

def _create_velocity_vectors(self) -> None:
n_values = [5, 6, 7, 1, 1]
spreads = [0.2, 0.4, 0.8, 1, 10]
2 changes: 1 addition & 1 deletion cloudnetpy/instruments/galileo.py
Original file line number Diff line number Diff line change
@@ -93,7 +93,7 @@ def galileo2nc(
galileo.add_nyquist_velocity(keymap)
galileo.add_site_geolocation()
valid_indices = galileo.add_zenith_and_azimuth_angles(
elevation_threshold=1,
elevation_threshold=1.1,
elevation_diff_threshold=0.1,
azimuth_diff_threshold=0.1,
)
12 changes: 9 additions & 3 deletions cloudnetpy/instruments/hatpro.py
Original file line number Diff line number Diff line change
@@ -31,7 +31,7 @@ def hatpro2l1c(
output_file: str,
site_meta: dict,
uuid: str | None = None,
date: str | None = None,
date: datetime.date | str | None = None,
) -> str:
"""Converts RPG HATPRO microwave radiometer data into Cloudnet Level 1c netCDF file.
@@ -45,7 +45,11 @@ def hatpro2l1c(
Returns:
UUID of the generated file.
"""
if isinstance(date, str):
date = datetime.date.fromisoformat(date)

coeff_files = site_meta.get("coefficientFiles")
time_offset = site_meta.get("time_offset")

try:
hatpro_raw = lev1_to_nc(
@@ -54,6 +58,8 @@ def hatpro2l1c(
output_file=output_file,
coeff_files=coeff_files,
instrument_config=site_meta,
date=date,
time_offset=time_offset,
)
except MissingInputData as err:
raise HatproDataError(str(err)) from err
@@ -73,7 +79,7 @@ def hatpro2l1c(
msg = "Timestamps are not increasing"
raise RuntimeError(msg)
dates = [
str(datetime.datetime.fromtimestamp(t, tz=datetime.timezone.utc).date())
datetime.datetime.fromtimestamp(t, tz=datetime.timezone.utc).date()
for t in timestamps
]
if len(set(dates)) != 1:
@@ -119,7 +125,7 @@ class HatproL1c:
def __init__(self, hatpro, site_meta: dict):
self.raw_data = hatpro.raw_data
self.data = hatpro.data
self.date = hatpro.date.split("-")
self.date = hatpro.date.isoformat().split("-")
self.site_meta = site_meta
self.instrument = HATPRO

15 changes: 13 additions & 2 deletions cloudnetpy/instruments/lufft.py
Original file line number Diff line number Diff line change
@@ -3,6 +3,7 @@
import logging

import netCDF4
import numpy as np
from numpy import ma

from cloudnetpy import utils
@@ -61,9 +62,19 @@ def _get_old_software_version(self) -> str | None:
msg = "No dataset found"
raise RuntimeError(msg)
version = self.dataset.software_version
if len(str(version)) > 4:
# In old files, the version is a single integer.
if isinstance(version, np.integer):
return str(version)
# In newer files, the version is a space-separated list: Operating
# system, FPGA, firmware, CloudDetectionMode (added in firmware 0.747).
if isinstance(version, str):
parts = version.split()
firmware = parts[2]
if firmware < "0.702":
return firmware
return None
return version
msg = f"Cannot determine version: {version}"
raise RuntimeError(msg)

def _get_nn(self) -> float | ma.MaskedArray:
nn1 = self._getvar("nn1", "NN1")
23 changes: 20 additions & 3 deletions cloudnetpy/instruments/mira.py
Original file line number Diff line number Diff line change
@@ -70,8 +70,11 @@ def mira2nc(
valid_ind = utils.remove_masked_blocks(mira.data["Zh"][:], limit=n_profiles)
mira.screen_time_indices(valid_ind)

# Empirical values, should be checked at some point...
snr_limit = -30 if mira.instrument == MIRA10 else -17
if "snr_limit" in site_meta and site_meta["snr_limit"] is not None:
snr_limit = site_meta["snr_limit"]
else:
# Empirical values, should be checked
snr_limit = -30 if mira.instrument == MIRA10 else -17

# Old MIRA files don't have angle variables.
if "elevation" not in mira.data:
@@ -87,9 +90,11 @@ def mira2nc(
mira.add_site_geolocation()
mira.add_radar_specific_variables()
valid_indices = mira.add_zenith_and_azimuth_angles(
elevation_threshold=1,
elevation_threshold=1.1,
elevation_diff_threshold=1e-6,
azimuth_diff_threshold=1e-3,
zenith_offset=site_meta.get("zenith_offset"),
azimuth_offset=site_meta.get("azimuth_offset"),
)
mira.screen_time_indices(valid_indices)
mira.add_height()
@@ -126,6 +131,8 @@ def screen_by_date(self, expected_date: str) -> None:
time_stamps = self.getvar("time")
valid_indices = []
for ind, timestamp in enumerate(time_stamps):
if not timestamp:
continue
date = "-".join(utils.seconds2date(timestamp, self.epoch)[:3])
if date == expected_date:
valid_indices.append(ind)
@@ -298,4 +305,14 @@ def _check_file_type(filetype: str) -> None:
long_name="Pulse Repetition Frequency",
units="Hz",
),
"zenith_offset": MetaData(
long_name="Zenith offset of the instrument",
units="degrees",
comment="Zenith offset applied.",
),
"azimuth_offset": MetaData(
long_name="Azimuth offset of the instrument (positive clockwise from north)",
units="degrees",
comment="Azimuth offset applied.",
),
}
10 changes: 10 additions & 0 deletions cloudnetpy/instruments/nc_radar.py
Original file line number Diff line number Diff line change
@@ -62,6 +62,7 @@ def screen_by_snr(self, snr_limit: float) -> None:
for cloudnet_array in self.data.values():
if cloudnet_array.data.ndim == 2:
cloudnet_array.mask_indices(ind)
self.append_data(float(snr_limit), "snr_limit")

def screen_using_top_gates_snr(self, snr_limit: float = 2) -> None:
"""Masks values where SNR is smaller than mean SNR of top gates."""
@@ -95,12 +96,21 @@ def add_zenith_and_azimuth_angles(
elevation_threshold: float,
elevation_diff_threshold: float,
azimuth_diff_threshold: float,
zenith_offset: float | None = None,
azimuth_offset: float | None = None,
) -> list:
"""Adds non-varying instrument zenith and azimuth angles and returns valid
time indices.
"""
elevation = self.data["elevation"].data
if zenith_offset is not None:
self.append_data(zenith_offset, "zenith_offset")
elevation -= zenith_offset

azimuth = self.data["azimuth_angle"].data
if azimuth_offset is not None:
self.append_data(azimuth_offset, "azimuth_offset")
azimuth += azimuth_offset

elevation_diff = ma.diff(elevation, prepend=elevation[1])
azimuth_diff = ma.diff(azimuth, prepend=azimuth[1])
1 change: 0 additions & 1 deletion cloudnetpy/instruments/rain_e_h3.py
Original file line number Diff line number Diff line change
@@ -31,7 +31,6 @@ def rain_e_h32nc(
UUID of the generated file.
Raises:
WeatherStationDataError : Unable to read the file.
ValidTimeStampError: No valid timestamps found.
"""
rain = RainEH3(site_meta)
Loading