v0.2.3 Release (#213)

* add top/bottom edge, median edge det, trained new model (#201) * bump version * silence perf logging * hotfix: tiled metadata * Add huggingface checkpoint downloads to load_model (#210) * hf checkpoint download * notebook update * rename weights to checkpoint * basic experiment widget * [FEAT] add logging for detection in workflow * added special logging for analytics (move, mill, det) * add fibsem image to DetectedFeatures * Gis time duration (#211) * changed time and put in update button * blank beam setting * cleaning up UI * gis final updates * fix fibsem image * [FEAT] Eucentric move update #197 * less info logging * Tab order (#212) fix tab ordering of UI * add fname logging for det * extra logging for stage / state * hfw milling threshold * fix thread issue temp * fix history from: v0.2-dev-fix-napari-patterns @Rohit-Kannachel (#215) * draw patterns on minimap * cmt * waffle figures * rm env file * add missing stages for protocol * update pattern logic * autoliftout stage positions * add unsafe flat to beam movement * system widget * img and movement widgets * detetction, milling, manipulator * move by coords * fix duplicate key for pattern selection * training upgrade * model configs * training / eval notebooks * threading * interactions toggle * V0.2 dev UI consistency (#219) * system widget * img and movement widgets * detetction, milling, manipulator * add model to demo * fixes --------- Co-authored-by: Patrick Cleeve <[email protected]> * V0.2 dev minimap upgrade (#222) * add all settings to minimap ui * thread tile collection, minimap * tile / stitching progress indicator * tile while collecting * formatting for button * updated undercut * add offset for det distance * fix thermo passes parameter * fix passes / config * added reproject to rotated positions * target identification from repojrected position * auto targeting * fixed logging, updated to json * bump version * imaging button orange * milling button orange * added napari notifications * notification signal imaging * no yielding in movement * no yielding in milling stage * fix move_based on_detection for new det, add point to detect_features * moving pattern update (#221) * moving pattern update * modified to not need an else block * improved logic to remove large else block * fixed stuff and removed comments oops hehe * fixed everything * solved world hunger --------- Co-authored-by: Patrick Cleeve <[email protected]> Co-authored-by: Rohit-Kannachel <[email protected]> * add version to ui * actually log final px position * fix px logging * test notebook * model evaluation improvements * refactor edge detection * cfg v4 waffle autolamella * updated wanbd logging * figures * cfg * eval notebook * tescan fixes * data engine image examples * improved eval code * system setup docs * connection * reorganisation * Imaging * eval data baseline * add features to ingore * movement * milling pt 1 * better aug * milling * working dataset aug * cfgs * update training code * eval / aug notebooks * eval plot update * V0.2.4 dev online labelling (#230) * start online labelling * fix saving data * disable sam for embedded labelling * V0.2 dev positions widget (#227) * minimap * append or overwrite positions file * loading positions on start up * system yaml * closing figures before making new ones * add position from external source * minor bug, smaller size --------- Co-authored-by: Patrick Cleeve <[email protected]> * reduce num updates minimap * milling * stage setup * upgrade needle tip det + test masks * testing masks * add vspot weld,change passes to numeric * vertical horseshoe pattern * bug: missing protocol values * add pattern type to protocol from stages * rename serial liftout test images * add additional det features, rename test images * rm plot * add multi points for multi patterns * refactor + add move_based_on_det * beam reset * manipulator * bugs: leaked code from merged branch * detection basic * minor edit * detection 2 * image setup * image patterns 2 * fix: tmp disable images after minimap movement * fix threaded movements * fix * detection * basic milling stages drawing napari * add default serial liftout positions * fix: serial liftout testing * fix: milling passes not set correctly (THERMO) * return flag, legend * rm unused files * edit docs * presets * moved into microscope * update evals * dev: ccd view * serial-test * add point projection to det func * minor fix * hotfix: projected position for demo microscope --------- Co-authored-by: Rohit-Kannachel <[email protected]> Co-authored-by: LucileNaegele <[email protected]> Co-authored-by: Rohit-Kannachel <[email protected]>
DeMarcoLab · Sep 8, 2023 · 246c449 · 246c449
1 parent d89f848
commit 246c449
Show file tree

Hide file tree

Showing 154 changed files with 9,208 additions and 3,350 deletions.
diff --git a/.gitignore b/.gitignore
@@ -52,4 +52,6 @@ fibsem/chat/secret.txt
 
 dist/*
 scratch/tile-images/*
-fibsem/log/*
+fibsem/log/*
+fibsem/db/fibsem.db
+fibsem/notebook.ipynb
diff --git a/docs/img/user_guide/connection/connect.png b/docs/img/user_guide/connection/connect.png
diff --git a/docs/img/user_guide/connection/connected.png b/docs/img/user_guide/connection/connected.png
diff --git a/docs/img/user_guide/connection/image_example.png b/docs/img/user_guide/connection/image_example.png
diff --git a/docs/img/user_guide/detection/detection_ex.png b/docs/img/user_guide/detection/detection_ex.png
diff --git a/docs/img/user_guide/imaging/image_tab.png b/docs/img/user_guide/imaging/image_tab.png
diff --git a/docs/img/user_guide/imaging/save_image.png b/docs/img/user_guide/imaging/save_image.png
diff --git a/docs/img/user_guide/manipulator/calibration.png b/docs/img/user_guide/manipulator/calibration.png
diff --git a/docs/img/user_guide/manipulator/control.png b/docs/img/user_guide/manipulator/control.png
diff --git a/docs/img/user_guide/milling/milling_stages.png b/docs/img/user_guide/milling/milling_stages.png
diff --git a/docs/img/user_guide/milling/milling_tab.png b/docs/img/user_guide/milling/milling_tab.png
diff --git a/docs/img/user_guide/milling/movement_1.png b/docs/img/user_guide/milling/movement_1.png
diff --git a/docs/img/user_guide/milling/rel_move.png b/docs/img/user_guide/milling/rel_move.png
diff --git a/docs/img/user_guide/milling/run_milling.png b/docs/img/user_guide/milling/run_milling.png
diff --git a/docs/img/user_guide/milling/unrel_move.png b/docs/img/user_guide/milling/unrel_move.png
diff --git a/docs/img/user_guide/movement/movement_tab.png b/docs/img/user_guide/movement/movement_tab.png
diff --git a/docs/img/user_guide/movement/positions_tab.png b/docs/img/user_guide/movement/positions_tab.png
diff --git a/docs/img/user_guide/setup/FIBSEM_DIAGRAM.png b/docs/img/user_guide/setup/FIBSEM_DIAGRAM.png
diff --git a/docs/img/user_guide/setup/config_tab.png b/docs/img/user_guide/setup/config_tab.png
diff --git a/docs/img/user_guide/setup/microscope_params.png b/docs/img/user_guide/setup/microscope_params.png
diff --git a/docs/img/user_guide/setup/stage_params.png b/docs/img/user_guide/setup/stage_params.png
diff --git a/docs/img/user_guide/setup/system_params.png b/docs/img/user_guide/setup/system_params.png
diff --git a/docs/img/user_guide/setup/user_params.png b/docs/img/user_guide/setup/user_params.png
diff --git a/docs/user_guide/connection.md b/docs/user_guide/connection.md
@@ -0,0 +1,21 @@
+# Connecting to a Microscope
+
+To begin connecting to a microscope, click on the connect tab. Here, you can connect to a microscope, load a configuration file, save a configuration file, and apply the settings to the microscope.
+
+Initially, none of the subsystems are connected and only the configuration tab is available. Please refer to the [System Configuration and Setup Guide](system_setup.md) for more information on the configuration tab. Make any changes there if necessary or continue to simply start with the default configuration.
+
+In the connect tab in the I.P. Address section, enter the I.P. address of the microscope, or localhost if the microscope is on the same computer. Then, click on the dropdown for manufacturer and select the manufacturer of the microscope. 
+
+![Connect Tab](../img/user_guide/connection/connect.png)
+
+Once done, click connect to microscope. This will connect to the microscope and load the default configuration. If the connection is successful, the button will display 'Microscope Connected' and the button will be highlighted in green. The other subsystem tabs will also become available.
+
+![Connected](../img/user_guide/connection/connected.png)
+
+If connection is unsuccessful, an error will arise. Please refer to the error messages for information on the error and troubleshooting solutions.
+
+Once connected to a microscope, other functionality is available. To take an image, click on the Image tab. Ensure the settings are as desired and click 'Acquire Image'. This will take an image and display it in the image viewer. Click 'Acquire All Images' to take both an ion and electron image. Please see the [imaging section](imaging.md) for more information on the Imaging.
+
+![Image Tab](../img/user_guide/connection/image_example.png)
+
+
diff --git a/docs/user_guide/detection.md b/docs/user_guide/detection.md
@@ -0,0 +1,53 @@
+# Feature Detection and Machine Learning Tools
+
+OpenFIBSEM hosts a variety of machine learning tools incorporated into the workflows for lamella preparation. This includes feature detection and classification through image segmentation and detection. 
+
+Through the use of detection and classification, OpenFIBSEM can be used to automate the process of lamella preparation. This is done by detecting the features of interest and classifying them into the relevant categories. This allows for the process to make decisions on its own regarding movement and milling ultimately reducing the need for human input. 
+
+OpenFIBSEM provides several baseline models for different methods and samples available on [huggingface](https://huggingface.co/patrickcleeve/openfibsem-baseline) for this purpose. These models have been finetuned for cryo-lamella preparation on different samples. You can also load from a local checkpoint, by specifying the full path. 
+
+To constantly improve the model, OpenFIBSEM also allows for user feedback to be incorporated into the model. This is done by allowing the user to verify the detection and classification of the features of interest. This feedback is then used to train the model to improve its accuracy. By default this data is logged to fibsem/log/data/ml. This can be changed in the config file.
+
+## Model selection
+
+To load a baseline from huggingface, specify the checkpoint name, and matching encoder. The encoder to use can be determined by the naming scheme of the checkpont, (e.g. openfibsem-baseline-34.pt -> encoder="resnet34", openfibsem-01-18.pt -> encoder="resnet18")
+
+A model can be loaded like so by using the load_model function found in fibsem/segmentation/model.py
+
+```python
+
+from fibsem.segmentation.model import load_model
+
+# load model checkpoint from huggingface (openfibsem-baseline-34.pt)
+model = load_model(checkpoint="openfibsem-baseline-34.pt", encoder="resnet34")
+
+# load base model (imagenet checkpoint)
+model = load_model(checkpoint=None, encoder="resnet34")
+
+# load a local checkpoint
+checkpoint_path = "path/to/model/checkpoint.pt" # Using model saved on disk
+model = load_model(checkpoint = checkpoint_path, encoder = "resnet18")
+
+```
+## Feature Detection in Automated Lamella Preparation
+
+Segmentation and feature detection is used in the automated lamellae preparation process to guide movement and milling. The regions of interest are segmented and detected, which are then used to control or make changes to position and milling parameters.
+
+![detection ex](../img/user_guide/detection/detection_ex.png)
+
+In the example shown above, the user is at the stage to mill the undercut for a lamella. The user is also supervising the workflow which ensures that the user has the final decision on the detection and feature position. 
+
+When repositioning to mill the undercut, the system runs feature detection to identify the position of the lamella centre to ensure accurate movement of the stage. In the image, the segmentation and detection of the lamella is shown. The feature of interest being the lamella centre is set as a point in the napari viewer.
+
+The user is then prompted to verify the detection and position of the feature. Here, if the position is invalid or incorrect, the user can simply drag the point to the correct location and press continue to proceed.
+
+This process of user verification is enabled when the workflow is run in supervised mode on the selected stages. When the workflow is run in unsupervised mode, the model is used to control the workflow without verification from the user. This is the ideal case when the model has been trained to a point user verification is not required, further reducing human involvement in the process.
+
+
+## Feedback and Model Training
+
+When in supervised mode, the user is prompted to verify the detection and position of the feature. When the user makes changes to the detection, metrics such as the new correct position, and how far the detection was from the user corrected position is used to analyse the performance and retrain the model if and when necessary.
+
+Based on whether the detection was correct or not, the images can be split into a training and validation dataset. The training dataset is used to train the model and the validation dataset is used to evaluate the model.
+
+OpenFIBSEM programs are constantly collecting this training and validation data enabling the model to be trained and improved over time. This allows for the model to be constantly improved and updated to the latest data.
diff --git a/docs/user_guide/guide.md b/docs/user_guide/guide.md
@@ -0,0 +1,11 @@
+# User Guide
+
+For more information on the subsystems, please refer to the respective sections in the user guide.
+
+- [Connection](connection.md)
+- [System Setup](system_setup.md)
+- [Imaging](imaging.md)
+- [Movement](movement.md)
+- [Milling](milling.md)
+- [Manipulator](manipulator.md)
+- [Feature Detection](detection.md)
diff --git a/docs/user_guide/imaging.md b/docs/user_guide/imaging.md
@@ -0,0 +1,29 @@
+# Imaging
+
+The imaging tab provides a suite of functionality to control the imaging process on both the electron and ion beam. This includes controlling image settings such as resolution and dwell time and processing methods such as auto contrast and autogamma.
+
+The settings for the detectors and the beam can also be altered here. There are also additional tools such as a scale bar and a ruler to take measurements. Details on these tools can be found [here](../features.md)
+
+The crosshair can be enabled or disabled in this tab as well. 
+
+![Image Tab](../img/user_guide/imaging/image_tab.png)
+
+When making changes to these settings, the settings will take effect when clicking 'Acquire Image'. The settings can be individually applied to the electron or ion beam or both simultaneously. The beam dropdown allows the user to select which beam to make changes for.  
+
+## Image Settings
+
+These settings are responsible for basic image parameters such as resolution, dwell time and horizontal field width. The resolution is specified in pixels, dwell time in micro-seconds and horizontal field width in microns. 
+
+There are also additional processing methods available, the AutoContrast and AutoGamma functions can be enabled or disabled as necessary. Enabling them activates the function for every image taken.
+
+Images can also be saved when required. Click the save image checkbox to enable image saving everytime a new image is taken. When enabled, additional fields will appear to allow the user to choose the save location and the file name.
+
+![save_image](../img/user_guide/imaging/save_image.png)
+
+## Detector Settings
+
+This section allows the user to alter and specify the detector settings available on the microscope. This includes the type and mode if available. The brightness and contrast can also be adjusted. To apply the changes, click 'Set Detector Settings' and then click Acquire Image(s) to take the image.
+
+## Beam Settings
+
+This section allows the user to modify the beam settings available on the microscope. This includes the  working distance (specified in mm), beam current (specified in picoAmps), voltage (kV) and beam shift in x and y (specified in microns). To apply the changes, click 'Set Beam Settings' and then click Acquire Image(s) to take the image.
diff --git a/docs/user_guide/manipulator.md b/docs/user_guide/manipulator.md
@@ -0,0 +1,24 @@
+# Manipulator
+
+The nanomanipulator or needle is a subsystem in some FIBSEM systems to allow for manipulation of the sample. This is typically used for lift-out and sample preparation. OpenFIBSEM allows for control of the manipulator through movement and saved positions. The manipulator section allows control for insertion and retraction along with movement in the x,y and z axis and rotation if supported.
+
+## Manipulator Calibration (TESCAN)
+
+For TESCAN systems, the manipulator needs to run a calibration setup upon first time usage on a microscope. This is done to correlate the named positions setup by TESCAN to the actual microscope coordinates, which allows reading the current position of the manipulator in terms of its name.
+
+If the manipulator is not calibrated, the subsystem will be disabled with the message indicating that the manipulator needs calibration. This is a one time operation but can be redone if necessary.
+
+![calibration](../img/user_guide/manipulator/calibration.png)
+
+To calibrate the manipulator, click on the tools menu and select "Manipulator Calibration". This will begin the process of calibration, during which, the manipulator moves rapidly between the "Parking", "Standby" and "Working" positions. Before calibrating it is recommend to move the stage down to ensure that the manipulator is free of obstructions so that no damage is done to the system. Follow the prompts on the screen to complete the calibration. It is completely automatic and requires no other user input. Once completed, it will be ready to use.
+
+## Manipulator Control
+
+The manipulator can be inserted and retracted as necessary. Inserting the manipulator brings it to the working area and retracting it brings it back to the standby position. 
+
+The manipulator can be controlled in the available axes, namely the x,y,z and rotation. The manipulator can be moved in the x,y,z axes by entering a value in the respective field and clicking the "Move" button. The movements made are relative moves. However, positions can be saved as named positions and moved to directly. 
+
+To save the current position as a named position, enter a name into the save position field and click "Save Position".
+To move to an available saved position, choose from the drop down menu and click "Go to Position" to go to the position.
+
+![control](../img/user_guide/manipulator/control.png)
diff --git a/docs/user_guide/milling.md b/docs/user_guide/milling.md
@@ -0,0 +1,81 @@
+# Milling
+
+The milling section allows the user to setup and control the milling process using the FIB. This includes setting up the milling patterns and specific milling settings.
+
+![milling_tab](../img/user_guide/milling/milling_tab.png)
+
+The milling process is done in milling stages. Each milling stage comprises of a pattern and the milling settings required for this pattern. The milling stages are then executed in the order of creation. A milling stage can be added by clicking the 'Add' button. An ion beam image must be taken first before a milling stage can be added. The stages are named in order of creation. A stage can be removed by pressing the remove button. A specific milling stage can be selected by choosing it from the milling stage drop down.
+
+## Milling Settings
+
+For each milling stage, there are milling settings associated with it, along with the pattern to be milled. The milling settings are specific to each manufacturer:
+
+TESCAN:
+
+* Milling rate (specified in mm<sup>3</sup> / A / s)
+* Dwell time (microseconds)
+* Spot size (microns)
+* Spacing (spot spacing)  
+* Preset  
+
+ThermoFisher:
+
+* Current (A)
+* Application File
+
+## Milling Patterns
+
+The pattern specifies the kind of pattern to be milled. These include simple shapes such as rectangles, circles and lines. However, OpenFIBSEM also includes a host of complex shapes ideal for lamella preparation. Each pattern has its own unqiue set of parameters that can be set. The full list of patterns is includes:
+
+* Rectangle
+* Line
+* Circle
+* Trench
+* Horseshoe
+* Undercut
+* Fiducial
+* Spot Weld
+* Micro Expansion
+* Waffle Notch
+* Clover
+* Triforce
+* Annulus
+* Bitmap (ThermoFisher Only)
+
+ By default, when changes are made to the pattern settings in the UI, the pattern displayed in the viewer is updated live. However, this can be disabled if necessary by unchecking the 'Live Update' checkbox.
+
+A pattern's position can be chosen relative to it's location on the ion image. This can be manually moved by entering coordinates in the centre x and centre y field. The coordinates of (0,0) is the centre of the image.
+
+Alternatively, the pattern can be moved anywhere within the image by pressing the *Shift* key and left-clicking on the image. If the pattern is out of bounds, it will return an out-of-bounds error and the pattern will remain where it was.
+
+## Multiple Stages
+
+Multiple stages can be setup at once. In this manner, once multiple stages are ready, clicking "Run Milling" will run all the stages sequentially without any more manual input. This avoids the repetitive nature of setting up and running stages individually.
+
+When there are multiple stages active, each seperate stage is coloured differently for identification. Changes can only be made to one stage at a time. To make changes to the stage, select the stage from the drop down menu and then make the relevant changes to the pattern or milling settings.
+
+![milling_stages](../img/user_guide/milling/milling_stages.png)
+
+### Moving Multiple Patterns
+
+*Shift* + *Ctrl* + Left-Click
+
+Moving multiple patterns simultaneously is also possible. To do this, press the *Shift* + *Ctrl* + Left-Click on a position in the ion image. By default, the checkbox "Keep Relative Orientation" is selected. This means when moving all the patterns, the relative orientation of the patterns to each other is kept the same. If this is not selected, all the patterns will be centred on the same new point.
+
+Moving multiple patterns uses the current selected pattern as the reference to move. I.e. the currently selected stage will move to the point chosen, while the other stages will move relative to this stage, retaining their relative position.
+
+![rel_move](../img/user_guide/milling/rel_move.png)
+
+To move all the patterns to a single location regardless of relative orientation, uncheck the "Keep Relative Orientation" checkbox. This will move all the patterns to the same location.
+
+![unrel_move](../img/user_guide/milling/unrel_move.png)
+
+## Run Milling
+
+Once satisfied with the milling and pattern settings and the stages are set as desired. Click "Run Milling" to perform milling. The milling stages will be performed in their numerical sequence order. When milling is running, the button will be highlighted orange
+
+![run_milling](../img/user_guide/milling/run_milling.png)
+
+**DURING MILLING, MOST OTHER FUNCTIONALITY IS DISABLED INCLUDING IMAGE ACQUISITION AND STAGE MOVEMENT**
+
+Once milling is complete, the button will return to its original colour and other functionality will be restored. The milling stages can be repeated or new ones can be added. The system will also automatically restore back to the original imaging current, and re-image the sample.
diff --git a/docs/user_guide/movement.md b/docs/user_guide/movement.md
@@ -0,0 +1,30 @@
+# Movement
+
+The movement tab provides functionality and control of the stage movement in the microscope system. This includes the ability to move the stage in the x, y and z directions as well as tilt and rotation if these are available in the microscope. 
+
+Additionally, the movement of the stage can be done by double clicking on a point in the image itself either in the electron or ion image (An image must be taken first). This will move the stage to the point clicked on and the point will now be centred in the view.
+
+![movement_tab](../img/user_guide/movement/movement_tab.png)
+
+## Stage Movement
+
+In the movement tab, the current coordinates are displayed. The coordinates are displayed in mm and angles in degrees where applicable. To move to a specific coordinate, enter the coordinates in the respective fileds and click 'Move to Position'. This will move the stage to the specified position. 
+
+The stage can also be tilted directly to be flat to each respective beam if necessary. To do this, click on either 'Move Flat to ELECTRON Beam' or 'Move Flat to ION Beam'. This will move the stage to be flat to the respective beam.
+
+By default, after every movement, an image is taken at the new position and the viewer is updated. However, this behaviour can be disabled if necessary. To disable this for the electron beam or ion beam or both, select the respective checkbox under 'Acquire images after moving'. The images will be taken with settings as specified in the image tab. For example, if autocontrast is enabled, the image will be autocontrasted every time a movement is made. This may be unnecessary and can be disabled if desired in the imaging tab.
+
+## Positions
+
+Specific positions can also be saved to be accessed or modified later. The positions tab allows control over this functionality
+
+![positions_tab](../img/user_guide/movement/positions_tab.png)
+
+To save a named position, simply enter a name for the position in the save position field and click 'Save position'. This will save the position and add it to the list of saved positions in the drop down. It's coordinates are displayed as well. To switch between saved positions, open the drop down and click on the saved position to go to and click 'Go To'. 
+
+A saved positions location can be updated if necessary. At the desired location, select the saved position from the drop down and click 'Update position'. This will update the position to the current location.
+
+A saved position can be removed if necessary by clicking 'Remove position'. This will remove the position from the list of saved positions.
+
+Saved positions can also be exported and imported if necessary. To export, click 'Export positions' and select a location to save the positions to. This will save the positions to a .yaml file. To import, click 'Import positions' and select the .yaml file to import. This will import the positions from the file and add them to the list of saved positions.
+