Categorical NB Food on Fork #104

atharva-kashyap · 2023-09-23T22:49:18Z

Description

This PR is adding a feature node Food on Fork that will allow for automatically detecting the presence of food on the fork. It is addressing the issue linked #43. It will output a confidence to a topic /food_on_fork and it represents the confidence with which the model is able to predict food on the fork.

Design Decisions

Outlined, in the readme, there is an overview of the models being used.

Testing procedure

Test the Training Procedure

Make sure to download the necessary testing files (ideally store them directly inside /ada_feeding_perception/ada_feeding_perception). Don't unzip the files. The code will unzip and process at runtime. The three datasets as linked below:
Change directory into /ada_feeding_perception/ada_feeding_perception/.
Run the food_on_fork_categorical_naive_bayes_training.py.
- This file takes in a few arguments. So, follow the command outlined below:
  - General format of the command: python food_on_fork_categorical_naive_bayes_training.py "<boolean>" "<dataset_zip>" --model_save_file "<name>"
    - The first argument (boolean) specifies whether or not to use the entire dataset for training. It also specifies whether or not a model should be saved.
    - Second argument (dataset_zip) specifies the location of the zip file that you have downloaded from the first step. Ideally, you should just store the downloaded zip file in ada_feeding_perception/ada_feeding_perception.
    - Third argument (name) is optional and is dependent on whether or not the first argument is true or false. If the first argument is true, the third argument is required, otherwise it is not required. This is the name that you want to provide for the model being saved
  - python food_on_fork_categorical_naive_bayes_training.py "False" "./train_test_data_no_hand_8-30-23.zip"
- <dataset_zip> specifies the dataset you downloaded.
- After running Dataset1, the expected output is as follows (confirm that this is the output you receive):

Train accuracy 0.999795605518651
Train confusion matrix
 [[4677    2]
 [   0 5106]]
Test accuracy 1.0
Test confusion matrix
 [[1253    0]
 [   0 1201]]

After running Datset2, the expected output is as follows:

Train accuracy 0.9998925193465176
Train confusion matrix
 [[4555    1]
 [   0 4748]]
Test accuracy 0.9996592844974447
Test confusion matrix
 [[1375    1]
 [   0 1559]]

After running Dataset3, the expected output is as follows:

Train accuracy 0.9997924234561495
Train confusion matrix
 [[4559    2]
 [   0 5074]]
Test accuracy 1.0
Test confusion matrix
 [[1371    0]
 [   0 1233]]

Note that parts of this procedure is also outlined here.

Test the model on the robot

Now that you have confirmed that you are able to mimic the training procedure, perform a model training on the full dataset. Run the following command to train on the full dataset: python food_on_fork_categorical_naive_bayes_training.py "True" "./train_test_data_no_hand_8-30-23.zip" --model_save_file "categorical_naive_bayes_without_hand_8-30-23.pkl". This should create a .pkl file, which is the model. Now either move this model into /ada_feeding_perception/model/ folder or use the model already present inside that folder. To be able to use this model, navigate to food_on_fork.yaml file and change the location of the model.

To test this model, follow these steps:

Position the robot such that the arm is in "resting position". Make sure to listen to /food_on_fork topic (ros2 topic echo /food_on_fork). If there is food on fork, you should notice values close to 1.0 and if there is no food on fork, you should notice values close to 0.0.
Run the FoF node and echo ros2 topic echo /food_on_fork and then visually test out by adding various food items onto the fork.
Then, reposition the arm into the "in-front of face" position, sit in the chair, and conduct the test outlined in Step 2.
Note: Try different food items. Try to test the extremes of adding food items as well. But, please don't get stuck on minor cases. There will still be cases where if the food is only partially on the fork and the prediction might be incorrect.

Before opening a pull request

Format your code using black formatter python3 -m black .
Run your code through pylint and address all warnings/errors. The only warnings that are acceptable to not address is TODOs that should be addressed in a future PR. From the top-level ada_feeding directory, run: pylint --recursive=y --rcfile=.pylintrc ..

Before Merging

Squash & Merge

…B model

ada_feeding_perception/README.md

ada_feeding_perception/ada_feeding_perception/food_on_fork.py

ada_feeding_perception/ada_feeding_perception/food_on_fork_categorical_naive_bayes_training.py

ada_feeding_perception/ada_feeding_perception/food_on_fork_logistic_reg_training.py

ada_feeding_perception/config/food_on_fork.yaml

amalnanavati

Mostly looks good, a few things should be changed but after you do that I envision is being ready to merge.

amalnanavati · 2023-11-20T18:20:26Z

ada_feeding_perception/README.md

+`/food_on_fork` topic. More documentation for the webapp can be found on its repository. 
+
+There are a couple models that can be used to determine Food on Fork. They are as follows:
+> Note that "frustum" is a 3D space around the fork


Not necessary, but consider linking a page explaining a frustrum, e.g., https://en.wikipedia.org/wiki/Frustum

amalnanavati · 2023-11-20T18:22:30Z

ada_feeding_perception/ada_feeding_perception/food_on_fork.py

+                    ParameterDescriptor(
+                        name="max_depth",
+                        type=ParameterType.PARAMETER_INTEGER,
+                        description="maximum depth to consider (note that 330 is approx distance "


Suggested change

description="maximum depth to consider (note that 330 is approx distance "

description="maximum depth to consider (note that 330mm is approx distance "

amalnanavati · 2023-11-20T18:22:37Z

ada_feeding_perception/ada_feeding_perception/food_on_fork.py

+                    ParameterDescriptor(
+                        name="min_depth",
+                        type=ParameterType.PARAMETER_INTEGER,
+                        description="minimum depth to consider (note that 330 is approx distance "


Suggested change

description="minimum depth to consider (note that 330 is approx distance "

description="minimum depth to consider (note that 330mm is approx distance "

amalnanavati · 2023-11-20T18:24:01Z

ada_feeding_perception/ada_feeding_perception/food_on_fork.py

+
+            # Tuning the rectangle
+            # depth_img_copy = np.copy(depth_img)
+            # cv.rectangle(depth_img_copy, (317, 238), (445, 322), (255, 0, 0))


If you want to keep this code in the file, plase explain what these numbers should refer to. e.g., consider making each of them variables that correspond to the parameter names, and then commenting out the whole block of code including those variable assignments

amalnanavati · 2023-11-20T18:26:38Z

ada_feeding_perception/ada_feeding_perception/food_on_fork.py

+
+        return np.count_nonzero(mask_img)
+
+    def predict_food_on_fork(


I don't think this is resolved. You still have two different parameters you pass in here. Instead, you should pass in a single parameter, x_test, which is an ndarray of size (1, num_features). Then, this function should just consist of

prediction_prob = self.model.predict_proba(x_test) return float(prediction_prob[0][1])

amalnanavati · 2023-11-20T18:31:13Z

ada_feeding_perception/ada_feeding_perception/food_on_fork.py

+    print("Running Food On Fork Node")
+    rclpy.init(args=args)
+    food_on_fork = FoodOnFork()
+    executor = MultiThreadedExecutor()


Since you only have one subscriber, you don't need the MultiThreadedExecutor. Please change to the SingleThreadedExecutor

amalnanavati · 2023-11-20T18:32:53Z

ada_feeding_perception/config/food_on_fork.yaml

+    # single feature or not
+    single_feature: False
+    # testing
+    test: False


It is convention for every file to end with a newline. Please add one here: https://stackoverflow.com/questions/729692/why-should-text-files-end-with-a-newline

amalnanavati · 2023-11-20T18:36:23Z

ada_feeding_perception/datasets/Crop_Images.py

+
+if __name__ == "__main__":
+    rosbags_abs_location = (
+        "/media/atharvak/oldFoodManip/rosbags_aligned_depth_10-17-23/"


5 years from now when someone is revisiting this code, they won't have access to this path to be able to check what the folder structure is.

What would be more helpful is: (a) allowing people to pass in an argument for the paths -- don't have it hardcoded into the file; (b) describe (either in a comment in this file, or in the README) what the expected directory structure is that this file needs (e.g., a folder with subfolders that each correspond to a single rosbag). That way, even if future programmers don't have access to the same dataset, they can recreate a dataset with the same directory structure, and then your code should work.

amalnanavati · 2023-11-20T18:37:11Z

ada_feeding_perception/datasets/ParseImagesFromRosbag.py

+
+if __name__ == "__main__":
+    rosbag_access_folder = (
+        "/media/atharvak/oldFoodManip/rosbags_aligned_depth_10-17-23/"


Same comment here re. hardcoded paths and lack of transperancy re. the expected directory structure.

amalnanavati · 2023-11-20T18:37:42Z

ada_feeding_perception/datasets/ParseImagesFromRosbag.py

Why do you need two separate files, one to parse images and the other to crop images? why not have one script that does both?

amalnanavati · 2024-04-06T17:32:51Z

Closed with #169

atharva-kashyap added 9 commits July 21, 2023 14:36

added FoF node

916733d

add FoF node!

8fa790d

made some changes based on comments

c0d635c

minor lint change

ec2d8aa

made some edits from comments

ab0ff95

after formatting

4da5c7a

new model working well

f9fe47b

merging with ros2-devel

ef95074

Updated with Food On Fork outputting confidence based on CategoricalN…

becbbca

…B model

atharva-kashyap mentioned this pull request Sep 23, 2023

Food On Fork Node V2 #40

Closed

21 tasks

atharva-kashyap self-assigned this Sep 23, 2023

atharva-kashyap added the enhancement New feature or request label Sep 23, 2023

atharva-kashyap changed the title ~~Atharvak/food on fork node v2~~ Categorical NB Food on Fork Sep 23, 2023

atharva-kashyap mentioned this pull request Sep 23, 2023

Atharvak/food on fork node #86

Closed

3 tasks

updates based on linting and documentation!

5f227db

atharva-kashyap requested a review from amalnanavati September 27, 2023 19:55

amalnanavati requested changes Sep 29, 2023

View reviewed changes

atharva-kashyap added 3 commits November 6, 2023 12:28

updated things from pr

100ae6c

Merge branch 'ros2-devel' into atharvak/food_on_fork_node_v2

4d38113

made all requested changes

5733184

atharva-kashyap requested a review from amalnanavati November 7, 2023 00:23

amalnanavati requested changes Nov 20, 2023

View reviewed changes

amalnanavati marked this pull request as ready for review November 20, 2023 18:38

amalnanavati closed this Apr 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Categorical NB Food on Fork #104

Categorical NB Food on Fork #104

atharva-kashyap commented Sep 23, 2023 •

edited

Loading

amalnanavati left a comment

amalnanavati Nov 20, 2023

amalnanavati Nov 20, 2023

amalnanavati Nov 20, 2023

amalnanavati Nov 20, 2023

amalnanavati Nov 20, 2023

amalnanavati Nov 20, 2023

amalnanavati Nov 20, 2023

amalnanavati Nov 20, 2023

amalnanavati Nov 20, 2023

amalnanavati Nov 20, 2023

amalnanavati commented Apr 6, 2024

	description="maximum depth to consider (note that 330 is approx distance "
	description="maximum depth to consider (note that 330mm is approx distance "

	description="minimum depth to consider (note that 330 is approx distance "
	description="minimum depth to consider (note that 330mm is approx distance "

Categorical NB Food on Fork #104

Categorical NB Food on Fork #104

Conversation

atharva-kashyap commented Sep 23, 2023 • edited Loading

Description

Design Decisions

Testing procedure

Test the Training Procedure

Test the model on the robot

Before opening a pull request

Before Merging

amalnanavati left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amalnanavati commented Apr 6, 2024

atharva-kashyap commented Sep 23, 2023 •

edited

Loading