0.4.0 Previewer #92

MasterVexillen · 2023-03-29T08:32:19Z

Change log

Changing tilt-series datatype from int (04d) to str
Modularised MGUI API according to respective plug-in. Old MGUI file not obsoleted yet.
Moved most parameter / metadata handling to MGUI functions for active plug-ins. Savu not migrated yet.
Added MGUI-no-show option to plug-MGUI functions.
Modified codebase to facilitate automation of MotionCor2 -> AreTomo workflow.

Note

Albeit thus named, issue with file-naming conventions in beam-shift data is not yet resolved in this PR.

… old magicgui.

…UI from old magicgui.

…m old magicgui.

…55/.local/bin:/home/wed13955/bin:/usr/local/go/bin:/root/go/bin:/usr/local/cuda/bin:/opt/lmod/modules/miniconda/4.10.3/bin:/opt/lmod/modules/miniconda/4.10.3/condabin:/usr/local/go/bin:/root/go/bin:/usr/local/cuda/bin:/usr/local/go/bin:/root/go/bin:/usr/local/cuda/bin:/sbin:/bin:/usr/bin:/usr/local/bin:/snap/bin:/ceph/groups/structbio/ot2rec-0.2.4/bin.

…e/wed13955/.local/bin:/home/wed13955/bin:/usr/local/go/bin:/root/go/bin:/usr/local/cuda/bin:/opt/lmod/modules/miniconda/4.10.3/bin:/opt/lmod/modules/miniconda/4.10.3/condabin:/usr/local/go/bin:/root/go/bin:/usr/local/cuda/bin:/usr/local/go/bin:/root/go/bin:/usr/local/cuda/bin:/sbin:/bin:/usr/bin:/usr/local/bin:/snap/bin:/ceph/groups/structbio/ot2rec-0.2.4/bin." This reverts commit 88a1a0b.

…UI from old magicgui.

…ed stack.

…d tomogram thickness to binned thickness.

…m old magicgui.

…MC2->AreTomo route.

…yet.

…cluded in commit f48a1e1.

elainehoml

Looks good overall, thanks Neville - a few bits and pieces to look at which are in the comments.
Main thing to consider is if we should separate out the previewer code into its own previewer.py because that would make it more consistent with how the other plugins are set up. it'll be a "meta plugin" but still works the same as the others and would be easier to maintain if it's separated out like the others.
I'll update the tests and add ot2rec_report to the previewer in a separate PR into this branch to keep this PR a manageable size

elainehoml · 2023-03-29T08:58:32Z

src/Ot2Rec/align.py

@@ -166,10 +167,17 @@ def create_stack_folders(self):

        # Create the folders and dictionary for future reference
        self._path_dict = {}
-        for curr_ts in self._process_list:
-            subfolder_path = f'{self.basis_folder}/{self.rootname}_{curr_ts:04}{self.suffix}'
+        if single_folder:


Does this mean that if you are processing images which are in a single folder (without the tilt series named subfolders), the processed data will live in a single folder as well rather than be organised into individual tilt series? We would have to account for this in the downstream plugins as well (maybe you have and I haven't gotten to that part yet...)

Yes, but only if so wished by the user. I've never had problems with this one as by default they are still organised into subfolders.

elainehoml · 2023-03-29T09:01:50Z

src/Ot2Rec/align.py

            'use_rawtlt': 1 if self.params['BatchRunTomo']['setup']['use_rawtlt'] else 0,
-            'pixel_size': self.params['BatchRunTomo']['setup']['pixel_size'],
+            'pixel_size': self.params['BatchRunTomo']['setup']['pixel_size'] * self.params['BatchRunTomo']['setup']['stack_bin_factor'],


Does this accidentally apply binning twice, since binning is already passed to batchruntomo with <aligned_bin_factor>

No. What it corrects is the wrong pixels size being passed into IMOD. This is because I've modified the code such that you can pre-bin the unaligned stacks first then do alignment. In that case, the old module would have the unbinned pixel size for alignment which is wrong.

The aligned_bin_factor is for the post-alignment stack (i.e. _.ali.mrc), whereas the stack_bin_factor is for the pre-aligned one (i.e. .st).

src/Ot2Rec/aretomo.py

elainehoml · 2023-03-29T09:06:53Z

src/Ot2Rec/aretomo.py

@@ -277,7 +283,7 @@ def _update_volz(args, aretomo_params):
                    in nm to automatically calculate VolZ. Currently sample_thickness \
                    = {args['sample_thickness']} and pixel_size = {args['pixel_size']}")
            aretomo_params.params["AreTomo_recon"]["volz"] = int(
-                (args["sample_thickness"] * args["pixel_size"] * 0.1) + 200   # factor of 0.1 because pixel_size in A
+                (10 * args["sample_thickness"] // args["pixel_size"]) + 200   # factor of 0.1 because pixel_size in A


Does args["pixel_size"] account for bin factors here?

No, because by definition VolZ is unbinned thickness in pixels

elainehoml · 2023-03-29T09:07:30Z

src/Ot2Rec/aretomo.py

@@ -30,6 +30,12 @@
 from . import magicgui as mgMod
 from . import params as prmMod
 from . import user_args as uaMod
+from . import mgui_imod_align as alignMGUI


I think it might be clearer to import this as align_args or something similar

elainehoml · 2023-03-29T09:15:17Z

src/Ot2Rec/main.py

+        meta.get_mc2_temp()
+        meta.get_acquisition_settings()
+
+    master_md_name = new_proj_params.project_name + '_master_md.yaml'


not for now but in the future maybe we should be consistent with one of either f strings or the + notation like here (my preference is fstrings but can decide later)

Yeah this is old stuff, I'm using f-strings recently

I think we could leave these minor stylistic polishing till later when the main codebase is more stable. There will be many of them in the modules as well, not just the main API.

setup.py

elainehoml · 2023-03-29T09:27:58Z

src/Ot2Rec/aretomo.py

@@ -290,7 +296,7 @@ def _update_volz(args, aretomo_params):
 def _create_stacks_with_imod(args):
    # Uses align to create the InMrc and AngFile in correct form
        try:
-            args_in_align = mgMod.get_args_align
+            args_in_align = alignMGUI.get_args_align


would this bring up the magicgui? in main.py you've used return_only=True into get_args

That would. However I'm not using this subroutine, but my old one directly from IMOD newstack (in align.py) instead so the GUI wouldn't come up like when you do command-line o2r.aretomo.run project 2.

elainehoml · 2023-03-29T09:39:36Z

src/Ot2Rec/main.py

+    mcMod.run(exclusive=False,
+              args_in=mc2_params)
+
+    time.sleep(2)


what's this for?

Leave 2 seconds between processes; I don't like automated to processes to run straight after one another.

elainehoml · 2023-03-29T10:43:30Z

src/Ot2Rec/mgui_mc2.py

+        max_iter=10,
+        patch_size=[5,5,20],
+        use_subgroups=True,
+        *,


what's the * for?

Just a separator

…tack_MGUI in aretomo.

…ipt.

…ct parsing in metadata.

…ewer.

EER metadata

MasterVexillen added 26 commits March 8, 2023 13:53

Parsing tilt-series indices as str in Metadata.

35d0837

Moved import operations from main to mgui. Separated import MGUI from…

3dae85c

… old magicgui.

Separated acquisition metadata from image metadata.

6ead747

Moved motioncorr metadata operations to mgui. Separated motioncorr MG…

e384aa7

…UI from old magicgui.

Removed forced 4-digit tilt-series index output from motioncorr.

51ae4f1

Modified motioncorr to run MC2 on a per-image (not per-TS) basis.

267d4e4

Removed 'jobs per gpu' option for MC2 - job chunking now obsolete.

ecd81be

Moved ctffind metadata operations to mgui. Separated ctffind MGUI fro…

303f2d0

…m old magicgui.

Modified mgui_mc2 to allow for default MC2 call from set PATH.

03c2d63

Fixed wrong exec_path input check in mgui_mc2.

aa621b2

Fixed wrong logger message tag in ctffind.

194dd83

Modified mgui_ctffind to allow for default CTFFind call from set PATH.

e06b017

Moved IMOD align metadata operations to mgui. Separated IMOD align MG…

ea12e75

…UI from old magicgui.

Removed debugging print statements from mgui_imod_align.

ac32754

Fixed wrong argument passing in motioncorr.

278c84e

Moved IMOD recon metadata operations to mgui. Separated IMOD recon MG…

ffa9088

…UI from old magicgui.

Fixed bug in IMOD align considering ratio between raw stack and align…

a0b2396

…ed stack.

Fixed mismatch in IMOD recon tomogram thickness. Changed reconstructe…

48aaf63

…d tomogram thickness to binned thickness.

Moved AreTomo metadata operations to mgui. Separated AreTomo MGUI fro…

bee13a2

…m old magicgui.

Corrected datatype of TS number in aretomo: int->str.

7c8d3fb

Fixed wrong thickness->VolZ conversion in aretomo.

71fefc8

Modified codebase to facilitate automatic full preprocessing through …

f48a1e1

…MC2->AreTomo route.

Added o2r.previewer entry point in setup. Version number NOT changed …

c533aa1

…yet.

Added back changes for O2R-Previewer in main API. Should have been in…

a9cedaa

…cluded in commit f48a1e1.

MasterVexillen added the enhancement New feature or request label Mar 29, 2023

MasterVexillen requested a review from elainehoml March 29, 2023 08:32

MasterVexillen self-assigned this Mar 29, 2023

Merge branch 'dev' into 0.4.0-beam-shift

eb9e22e

Added back mgui_auto_aretomo module file.

166b98b

elainehoml reviewed Mar 29, 2023

View reviewed changes

MasterVexillen and others added 28 commits March 29, 2023 14:27

Changed MGUI module load in aretomo from old magicgui to mgui_aretomo.

1f66f27

Changed mgui_imod_align module load alias from alignMGUI to imod_news…

fe2f2be

…tack_MGUI in aretomo.

Migrated previewer from main API to its own module. Updated setup scr…

f312bec

…ipt.

Fixed typo in previewer.

a55e233

Rename: mgui_auto_aretomo.py -> mgui_previewer.py

9dffad9

Added back aretomo_path as aretomo MGUI option.

4b07825

Removed redundant mgui_previewer import in main (previewer modularised).

fa42293

Fixed typo in previewer. Added default aretomo_path in previewer.

f9d8607

Added previewer into default package list in __init__.

f51a6b4

Fixed datatype for tilt-series index in aretomo.

0cbca9f

black formatting

03657d1

add ot2rec-report dependency

f2b5791

suppoer mdf 0.0.7 camel case mdoc fields

8118996

add ot2rec report

d18ec1d

pin mdocfile version

e3e0b89

aln filenames correction

25a13a8

enable aretomo path specification

97d6499

Added xmltodict as dependency in setup script.

edce9d3

Changed EER metadata scraping method from calling IMOD header to dire…

2790e2d

…ct parsing in metadata.

DEBUG: removed breakpoint in metadata.

16af159

Added progress bar for metadata scraping in metadata.

be8a7a3

Added log messages for program initiation and metadata scraping.

b132fc9

Added plain HTML output for report in previewer.

cac4323

Redesigned logger object definitions in logger.

4db1880

Updated logger for O2R-Previewer components up to motioncor.

240f642

Updated logger for O2R-Previewer components up to report generation.

cded059

Changed label (Lamella thickness -> Tomogram thickness) in mgui_previ…

563f0dc

…ewer.

Merge pull request #93 from rosalindfranklininstitute/eer-metadata

a0febfe

EER metadata

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

0.4.0 Previewer #92

0.4.0 Previewer #92

MasterVexillen commented Mar 29, 2023 •

edited

Loading

elainehoml left a comment

elainehoml Mar 29, 2023

MasterVexillen Mar 29, 2023

elainehoml Mar 29, 2023

MasterVexillen Mar 29, 2023

MasterVexillen Mar 29, 2023

elainehoml Mar 29, 2023

MasterVexillen Mar 29, 2023

elainehoml Mar 29, 2023

elainehoml Mar 29, 2023

MasterVexillen Mar 29, 2023

MasterVexillen Mar 29, 2023

elainehoml Mar 29, 2023

MasterVexillen Mar 29, 2023

elainehoml Mar 29, 2023

MasterVexillen Mar 29, 2023

elainehoml Mar 29, 2023

MasterVexillen Mar 29, 2023

0.4.0 Previewer #92

Are you sure you want to change the base?

0.4.0 Previewer #92

Conversation

MasterVexillen commented Mar 29, 2023 • edited Loading

Change log

Note

elainehoml left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MasterVexillen commented Mar 29, 2023 •

edited

Loading