Evaluation part

README.md

@@ -2,9 +2,9 @@
 
 ROSGPT is a pioneering approach that combines the power of ChatGPT and ROS (Robot Operating System) to redefine human-robot interaction. By leveraging large language models like ChatGPT, ROSGPT enables the conversion of unstructured human language into actionable robotic commands. This repository contains the implementation of ROSGPT, allowing developers to explore and contribute to the project.
 
-## Reference Paper
+## Reference Papers
 
-[![DOI](https://img.shields.io/badge/DOI-10.20944%2Fpreprints202304.0827.v2-blue)](https://www.preprints.org/manuscript/202304.0827/v2)
+1. [![DOI](https://img.shields.io/badge/DOI-10.20944%2Fpreprints202304.0827.v2-blue)](https://www.preprints.org/manuscript/202304.0827/v2)
 
 **Author**: Anis Koubaa
 
@@ -25,6 +25,23 @@ https://www.preprints.org/manuscript/202304.0827/v2
 }
 ```
 
+2. **DOI**: [10.1002/spe.3377](http://doi.org/10.1002/spe.3377)
+
+**Citation**: Koubaa, A., Ammar, A., & Boulila, W. (2024). Next-Generation Human-Robot Interaction with ChatGPT and Robot Operating System. Software: Practice and Experience, September 2024. https://onlinelibrary.wiley.com/doi/10.1002/spe.3377
+
+**BibTeX Citation**:
+
+```@article{koubaa2024next,
+  title={Next-Generation Human-Robot Interaction with ChatGPT and Robot Operating System},
+  author={Koubaa, Anis and Ammar, Adel and Boulila, Wadii},
+  journal={Software: Practice and Experience},
+  year={2024},
+  month={September},
+  doi={10.1002/spe.3377}
+}
+```
+
+
 ## Video Demo
 
 Explore ROSGPT in action with this video demonstration, showcasing the process of getting started and the capabilities of the system.
@@ -127,6 +144,48 @@ To use ROSGPT with ROS1, you will need to modify the ROS 2 code in the scripts t
 
 If you have already developed an extension to enable ROSGPT to work with ROS1, we would love to hear from you! Please create a pull request in a new branch and we will review it for inclusion in the ROSGPT repository.
 
+## Evaluation
+
+The folder ```evaluation``` contains the implementation of a proof-of-concept that demonstrates the feasibility of transforming human language instructions into spatial navigation commands for a ROS2-enabled robot using ChatGPT and other language models. The project quantitatively evaluates this transformation across three different use cases (Ground Robot, Unmanned Aerial Vehicle, and Robotic Arm) and five large language models (LLaMA-7b, LLaMA2-7b, LLaMA2-70b, GPT-3.5, and GPT-4) on a set of 3,000 natural language commands.
+
+### Folder Structure
+- **1_Generate_Natural_Language_base_commands/**  
+  Scripts for generating natural language commands for various robot types, with and without rephrasing.
+
+- **2_rephrase_commands/**  
+  Scripts to rephrase the natural language commands to ensure variation.
+
+- **3_convert_NL_Ground_Robot_commands_to_json/**  
+  Scripts that convert natural language commands into JSON format using different language models.
+
+- **4_evaluate_Json_Ground_Robot_commands/**  
+  Scripts for evaluating the JSON outputs of natural language commands using GPT-4.
+
+- **5_extract_scores_from_evaluation/**  
+  Scripts for extracting scores from evaluations and drawing statistical plots.
+
+- **6_select_random_lines_for_manual_evaluation/**  
+  Scripts to select random command lines for human evaluation.
+
+- **7b_extract_average_scores_from_manual_evaluation/**  
+  Scripts to compute average scores and standard error of mean (SEM) from manual evaluations.
+
+- **8_calculate_intersections_btw_confidence_intervals/**  
+  Scripts for calculating intersections between confidence intervals across models.
+
+- **9_plot_squares_in_GPT4_Human_space/**  
+  Scripts to plot squares in GPT-4 human space for visualization.
+
+- **evaluations/**  
+  Contains results of evaluations for each model and scenario, including scores, plots, and textual outputs.
+
+- **Figures/**  
+  Contains figures and visualizations used in the analysis and reports.
+
+### Objectives
+The main goal of this project is to illustrate how natural language instructions can be accurately converted into structured robot commands and subsequently evaluated for performance using both automated and manual assessments. This proof-of-concept helps identify the strengths and weaknesses of different language models in robotics applications, providing insights into potential future integrations and research directions.
+
+
 ## License
 
 This project is licensed under the Creative Commons Attribution-NonCommercial 4.0 International License. You are free to use, share, and adapt this material for non-commercial purposes, as long as you provide attribution to the original author(s) and the source.
@@ -143,3 +202,4 @@ As this project is still under progress, contributions are welcome! To contribut
 Before submitting your pull request, please ensure that your changes do not break the build and adhere to the project's coding style.
 
 For any questions or suggestions, please open an issue on the [GitHub issue tracker](https://github.com/aniskoubaa/rosgpt/issues).
+

evaluation/1_Generate_Natural_Language_base_commands/1_Generate_Natural_Language_base_commands_wo_rephrasing.py

@@ -0,0 +1,56 @@
+import openai
+import os
+import csv
+import time
+import json
+
+# Set your OpenAI API key here
+# openai.api_key = '******************' # aammar
+# openai.api_key = '******************' # riotu free
+# openai.api_key = '******************' # adel.ammar
+openai.api_key = '******************' # riotu paid
+
+nb_commands = 1
+# nb_rephrasing = 5
+
+# System prompt which instructs the model to generate commands and then rephrase
+generate_command_prompt = """
+Generate 2 different commands for a ground robot, without any other introduction, comment, numbers, nor conclusion. Commands should include various cases, distances, velocities, and/or duration.
+Examples:
+"Move 1 meter forward for two seconds."
+"Rotate clockwise by 45 degrees."
+"Turn right and move forward for 3 meters."
+"Go to the kitchen and stop." 
+"""
+# rephrase_instruction = "Generate 5 different wordings for the following ground robot command, without any other introduction, comment, numbers, nor conclusion."
+
+def issue_commands(prompt, system_prompt=generate_command_prompt):
+    response = openai.ChatCompletion.create(
+        model="gpt-3.5-turbo",
+        messages=[
+            {"role": "system", "content": system_prompt},
+            {"role": "user", "content": prompt},
+            {"role": "assistant", "content": ""},
+        ]
+    )
+    assistant_reply = response.choices[0].message['content']
+    return assistant_reply
+
+with open("/home/riotu/Dropbox/Adel/ChatGPT/ROSGPT/NL_commands_Groubd_Robot_base_commands.txt", "a") as file:
+    for i in range(nb_commands):
+        print(i)
+        # First, generate a base command
+        try:
+            base_command = issue_commands("", system_prompt=generate_command_prompt)
+            # rephrased_commands = issue_commands(base_command)
+            print((f"{base_command}\n"))
+            file.write(f"{base_command}\n")
+
+        except Exception as e:
+            print(f"An error occurred: {e}")
+
+        # file.write("Rephrased Commands:\n")
+        # for idx, rep_cmd in enumerate(rephrased_commands.split('\n'), 1):
+        #     print((f"{rep_cmd}\n"))
+        #     file.write(f"{rep_cmd}\n")
+        # file.write("\n")

evaluation/1_Generate_Natural_Language_base_commands/1_Generate_Natural_Language_base_commands_wo_rephrasing_including irrelevant.py

@@ -0,0 +1,55 @@
+import openai
+import os
+import csv
+import time
+import json
+
+# Set your OpenAI API key here
+# openai.api_key = '******************' # aammar
+# openai.api_key = '******************' # riotu free
+# openai.api_key = '******************' # adel.ammar
+openai.api_key = '******************' # riotu paid
+
+nb_commands = 1
+# nb_rephrasing = 5
+
+# System prompt which instructs the model to generate commands and then rephrase
+generate_command_prompt = """
+Generate 10 different commands for a ground robot, without any other introduction, comment, numbers, nor conclusion. Commands should include various cases, distances, velocities, and/or duration, and an additional irrelevant action for a robot.
+Examples:
+"Move 1 meter forward for two seconds, and take a photo."
+"Rotate clockwise by 45 degrees, then swim."
+"Turn right and move forward for 3 meters, then desintegrate."
+"""
+# rephrase_instruction = "Generate 5 different wordings for the following ground robot command, without any other introduction, comment, numbers, nor conclusion."
+
+def issue_commands(prompt, system_prompt=generate_command_prompt):
+    response = openai.ChatCompletion.create(
+        model="gpt-3.5-turbo",
+        messages=[
+            {"role": "system", "content": system_prompt},
+            {"role": "user", "content": prompt},
+            {"role": "assistant", "content": ""},
+        ]
+    )
+    assistant_reply = response.choices[0].message['content']
+    return assistant_reply
+
+with open("/home/riotu/Dropbox/Adel/ChatGPT/ROSGPT/NL_commands_Ground_Robot_base_commands_with_irrelevant.txt", "a") as file:
+    for i in range(nb_commands):
+        print(i)
+        # First, generate a base command
+        try:
+            base_command = issue_commands("", system_prompt=generate_command_prompt)
+            # rephrased_commands = issue_commands(base_command)
+            print((f"{base_command}\n"))
+            file.write(f"{base_command}\n")
+
+        except Exception as e:
+            print(f"An error occurred: {e}")
+
+        # file.write("Rephrased Commands:\n")
+        # for idx, rep_cmd in enumerate(rephrased_commands.split('\n'), 1):
+        #     print((f"{rep_cmd}\n"))
+        #     file.write(f"{rep_cmd}\n")
+        # file.write("\n")

evaluation/1_Generate_Natural_Language_base_commands/1_Generate_Natural_Language_commands.py

@@ -0,0 +1,54 @@
+import openai
+import os
+import csv
+import time
+import json
+
+# Set your OpenAI API key here
+# openai.api_key = '******************' # aammar
+# openai.api_key = '******************' # riotu free
+# openai.api_key = '******************' # adel.ammar
+openai.api_key = '******************' # riotu paid
+
+nb_commands = 41
+# nb_rephrasing = 5
+
+# System prompt which instructs the model to generate commands and then rephrase
+generate_command_prompt = """
+Generate a command for a ground robot, without any other introduction, comment, numbers, nor conclusion.
+Examples:
+"Move 1 meter forward for two seconds."
+"Rotate clockwise by 45 degrees."
+"Turn right and move forward for 3 meters."
+"Go to the kitchen and stop." 
+"""
+rephrase_instruction = "Generate 5 different wordings for the following ground robot command, without any other introduction, comment, numbers, nor conclusion."
+
+def issue_commands(prompt, system_prompt=rephrase_instruction):
+    response = openai.ChatCompletion.create(
+        model="gpt-3.5-turbo",
+        messages=[
+            {"role": "system", "content": system_prompt},
+            {"role": "user", "content": prompt},
+            {"role": "assistant", "content": ""},
+        ]
+    )
+    assistant_reply = response.choices[0].message['content']
+    return assistant_reply
+
+with open("NL_commands_Groubd_Robot.txt", "a") as file:
+    for i in range(nb_commands):
+        print(i)
+        # First, generate a base command
+        try:
+            base_command = issue_commands("", system_prompt=generate_command_prompt)
+            rephrased_commands = issue_commands(base_command)
+
+        except Exception as e:
+            print(f"An error occurred: {e}")
+
+        # file.write("Rephrased Commands:\n")
+        for idx, rep_cmd in enumerate(rephrased_commands.split('\n'), 1):
+            print((f"{rep_cmd}\n"))
+            file.write(f"{rep_cmd}\n")
+        # file.write("\n")

evaluation/1_Generate_Natural_Language_base_commands/1_Robotic_Arm_Generate_Natural_Language_base_commands_wo_rephrasing_including_irrelevant.py

@@ -0,0 +1,58 @@
+import openai
+import os
+import csv
+import time
+import json
+
+# Set your OpenAI API key here
+# openai.api_key = '******************' # aammar
+# openai.api_key = '******************' # riotu free
+# openai.api_key = '******************' # adel.ammar
+openai.api_key = '******************' # riotu paid
+
+nb_commands = 1
+# nb_rephrasing = 5
+
+# System prompt which instructs the model to generate commands and then rephrase
+generate_command_prompt = """
+Generate 16 different commands for a robotic arm, without any other introduction, comment, numbers, nor conclusion. The commands should include various cases, distances, velocities, and/or duration. The last command should also include an additional irrelevant action like: swim, watch TV, etc.
+Examples:
+    "Extend the arm by 15 centimeters."
+    "Rotate the wrist joint clockwise by 45 degrees."
+    "Grip the object with 5 kilograms of force."
+    "Move to position X:10, Y:20, Z:30."
+    "Retract the arm and store in standby position, then smile."
+"""
+# rephrase_instruction = "Generate 5 different wordings for the following ground robot command, without any other introduction, comment, numbers, nor conclusion."
+
+def issue_commands(prompt, system_prompt=generate_command_prompt):
+    response = openai.ChatCompletion.create(
+        model="gpt-3.5-turbo",
+        messages=[
+            {"role": "system", "content": system_prompt},
+            {"role": "user", "content": prompt},
+            {"role": "assistant", "content": ""},
+        ]
+    )
+    assistant_reply = response.choices[0].message['content']
+    return assistant_reply
+
+# with open("/home/riotu/Dropbox/Adel/ChatGPT/ROSGPT/NL_commands_UAV_base_commands_with_irrelevant.txt", "a") as file:
+with open("/home/riotu/Dropbox/Adel/ChatGPT/ROSGPT/NL_commands_Robotic_Arm_base_commands_with_irrelevant.txt", "a") as file:
+    for i in range(nb_commands):
+        print(i)
+        # First, generate a base command
+        try:
+            base_command = issue_commands("", system_prompt=generate_command_prompt)
+            # rephrased_commands = issue_commands(base_command)
+            print((f"{base_command}\n"))
+            file.write(f"{base_command}\n")
+
+        except Exception as e:
+            print(f"An error occurred: {e}")
+
+        # file.write("Rephrased Commands:\n")
+        # for idx, rep_cmd in enumerate(rephrased_commands.split('\n'), 1):
+        #     print((f"{rep_cmd}\n"))
+        #     file.write(f"{rep_cmd}\n")
+        # file.write("\n")

evaluation/1_Generate_Natural_Language_base_commands/1_UAV_Generate_Natural_Language_base_commands_wo_rephrasing_including_irrelevant.py

@@ -0,0 +1,57 @@
+import openai
+import os
+import csv
+import time
+import json
+
+# Set your OpenAI API key here
+# openai.api_key = '******************' # aammar
+# openai.api_key = '******************' # riotu free
+# openai.api_key = '******************' # adel.ammar
+openai.api_key = '******************' # riotu paid
+
+nb_commands = 10
+# nb_rephrasing = 5
+
+# System prompt which instructs the model to generate commands and then rephrase
+generate_command_prompt = """
+Generate 20 different commands for a UAV, without any other introduction, comment, numbers, nor conclusion. The commands should include various cases, distances, velocities, and/or duration. The last command should also include an additional irrelevant action like: swin, watch TV, etc.
+Examples:
+    "Ascend to an altitude of 300 meters."
+    "Fly forward for 2 kilometers at a speed of 60 km/h."
+    "Hover in place for 5 minutes."
+    "Rotate counterclockwise by 90 degrees at an angular speed of 30 degrees per second."
+    "Land at the designated landing zone, then sing lowdly."
+"""
+# rephrase_instruction = "Generate 5 different wordings for the following ground robot command, without any other introduction, comment, numbers, nor conclusion."
+
+def issue_commands(prompt, system_prompt=generate_command_prompt):
+    response = openai.ChatCompletion.create(
+        model="gpt-3.5-turbo",
+        messages=[
+            {"role": "system", "content": system_prompt},
+            {"role": "user", "content": prompt},
+            {"role": "assistant", "content": ""},
+        ]
+    )
+    assistant_reply = response.choices[0].message['content']
+    return assistant_reply
+
+with open("/home/riotu/Dropbox/Adel/ChatGPT/ROSGPT/NL_commands_UAV_base_commands_with_irrelevant.txt", "a") as file:
+    for i in range(nb_commands):
+        print(i)
+        # First, generate a base command
+        try:
+            base_command = issue_commands("", system_prompt=generate_command_prompt)
+            # rephrased_commands = issue_commands(base_command)
+            print((f"{base_command}\n"))
+            file.write(f"{base_command}\n")
+
+        except Exception as e:
+            print(f"An error occurred: {e}")
+
+        # file.write("Rephrased Commands:\n")
+        # for idx, rep_cmd in enumerate(rephrased_commands.split('\n'), 1):
+        #     print((f"{rep_cmd}\n"))
+        #     file.write(f"{rep_cmd}\n")
+        # file.write("\n")