Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for TOF imaging #309

Merged
merged 11 commits into from
Sep 19, 2024
Merged

Add support for TOF imaging #309

merged 11 commits into from
Sep 19, 2024

Conversation

KedoKudo
Copy link
Contributor

@KedoKudo KedoKudo commented Sep 5, 2024

This PR introduces this following changes:

  • Disable memmap (the instrument team reported that it is slowing down the reading)
  • Both .tiff and .tif are considered valid extension for TIFF input format now
  • During normalization, one can specify darks=None to assume perfect dark fields (all dark fields are zero)
  • The data loader now attempts to extract rotation angles from files, returning None if extraction fails. For files with partially missing rotation angles, the loader will return np.nan in the rotation angle list, allowing users to handle or sanitize the missing values as needed. While files must still adhere to the correct naming pattern (or contain the necessary metadata) for auto reduction, the auto reconstruction process will no longer fail if rotation angle extraction is unsuccessful.
  • The tilt correction functionality now allows users to specify the rotation center for tilt. By default, iMars3D is assuming the center of the image is the rotation center for tilt correction. Users can overwrite this by providing the actual rotation center as a tuple to tilt related functions.
  • Add a new function to calculate the chunk size for tqdm multi-processing wrapper. This is to resolve the warning about too many chunks when processing large number of images.

EWM story: 6746

Copy link

codecov bot commented Sep 16, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.73%. Comparing base (8e2b4e0) to head (6d79fbf).
Report is 25 commits behind head on next.

Additional details and impacted files
@@            Coverage Diff             @@
##             next     #309      +/-   ##
==========================================
- Coverage   94.62%   93.73%   -0.90%     
==========================================
  Files          21       21              
  Lines        1266     1277      +11     
==========================================
- Hits         1198     1197       -1     
- Misses         68       80      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@KedoKudo KedoKudo changed the title [WIP] Add support for TOF imaging Add support for TOF imaging Sep 17, 2024
@KedoKudo KedoKudo self-assigned this Sep 17, 2024
Copy link
Collaborator

@peterfpeterson peterfpeterson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small comments and changes. One bigger question is if you use mypy or just have the type annotations?

src/imars3d/backend/dataio/data.py Outdated Show resolved Hide resolved
src/imars3d/backend/dataio/data.py Show resolved Hide resolved
if all(matches):
logger.info("Using rotation angles from filenames.")
rotation_angles = np.array([float(".".join(m.groups())) for m in matches])
regex = r"\d{8}_\S*_\d{4}_(?P<deg>\d{3})_(?P<dec>\d{3})_\d*\.(?:tiff?|fits)"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should add tests for this function to press on this regex a lot harder

src/imars3d/backend/dataio/data.py Show resolved Hide resolved
@@ -103,6 +103,7 @@ def calculate_dissimilarity(
tilt: float,
image0: np.ndarray,
image1: np.ndarray,
center: Optional[Tuple[Union[float, int], Union[float, int]]] = None,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be more readable if you created a custom type. Maybe

type Number = Union[float, int]
type Pair = Tuple[Number, Number]
...
center: Optional[Pair] = None,

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it will be easier once we make 3.11 the minimum version of python as we can write it as Optional[Tuple[float|int, float|int]].

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that the compound typing you have is harder to read. You can keep it how you want, but center: Optional[Tuple[Number, Number]] = None is significantly clearer to me.

src/imars3d/backend/diagnostics/tilt.py Show resolved Hide resolved
src/imars3d/backend/util/functions.py Outdated Show resolved Hide resolved
tests/unit/backend/dataio/test_data.py Show resolved Hide resolved
@@ -124,6 +124,24 @@ def test_normalization_bright_dark():
assert diff < 0.01


def test_normalization_no_darks():
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know about numpy.testing?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am assuming you are asking to use np.testing.assert_all_close instead of computing the average difference here. They are similar in function, but I prefer the explicit one here over the one provided by numpy

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a thing to point out as existing

KedoKudo and others added 3 commits September 19, 2024 11:55
provide file type in error messages as suggested

Co-authored-by: Pete Peterson <[email protected]>
use consistent type hinting

Co-authored-by: Pete Peterson <[email protected]>
@KedoKudo
Copy link
Contributor Author

Small comments and changes. One bigger question is if you use mypy or just have the type annotations?

Just the type annotations. We are not using mypy for iMars3D.

Copy link
Collaborator

@peterfpeterson peterfpeterson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think adding mypy to this repository will make you have to work a bit more on your type annotations. The code looks reasonable.

@KedoKudo KedoKudo merged commit d0f0908 into next Sep 19, 2024
6 checks passed
@KedoKudo KedoKudo deleted the support_tof branch September 19, 2024 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants