diff --git a/README.cn.md b/README.cn.md new file mode 100644 index 0000000..51dedbf --- /dev/null +++ b/README.cn.md @@ -0,0 +1,52 @@ +# mnlm +机器人臂演示的源代码（参见 https://www.bilibili.com/video/BV1ub4y1T7Jt/）。 + +[![图片替代文本](./images/screen.png)](https://www.bilibili.com/video/BV1ub4y1T7Jt/?vd_source=08295b5b4b3c5ece73fb91e3a54d202a) + +## 构建并启动Docker容器 + +1. 安装 [Docker](https://docs.docker.com/get-docker/) 和 [Docker Compose](https://docs.docker.com/compose/install/)。 +```bash +docker-compose up --build -d +``` + +此命令将为服务器端构建docker镜像，并启动docker容器。 +用户也可以通过浏览器访问 http://localhost:8080/vnc.html 来访问模拟环境。 + +[![IMAGE ALT TEXT HERE](./images/novnc.png)] + +1. 登录到docker容器。 +如果您使用vscode，您可以安装 Dev Container 扩展并打开正在运行的容器。否则，您可以通过运行以下命令登录到docker容器： +``` +docker exec -it mnln-ros_dev_env-1 /bin/bash +``` + +2. 启动ROS2模拟。 + +```bash +cd /home/small-thinking/mnlm/mnlm/robot/robot_arm_ws +``` + +``` +colcon build --symlink-install ; source install/setup.bash ; ros2 launch robot_arm robot_arm.launch.py +``` + +您应该会看到服务器端程序启动。并且您可以访问 http://localhost:8080/vnc.html 来看到Gazebo Fortress模拟环境。 + +![IMAGE ALT TEXT HERE](./images/gazebo.png) + +3. 在您的宿主机上。您可以运行命令以启动基于声音的UI。 + +进入项目文件夹： +```bash +cd mnlm/client/gpt_control +``` + +```bash +python assistant.py +``` +然后您可以看到客户端启动，并且您将被提示告诉机器人要做什么。 + +![IMAGE ALT TEXT HERE](./images/voice.png) + + diff --git a/README.md b/README.md index 7a3405c..8b73627 100644 --- a/README.md +++ b/README.md @@ -1,103 +1,52 @@ -# mnlm -Source code of robotic arm demo (see https://www.bilibili.com/video/BV1ub4y1T7Jt/). +# mnlm ([中文文档](README.cn.md)) -TODOs: -1. Restructure the data format of the robot arm servo control. +Source code of robotic arm demo (see https://www.bilibili.com/video/BV1ub4y1T7Jt/). [![IMAGE ALT TEXT HERE](./images/screen.png)](https://www.bilibili.com/video/BV1ub4y1T7Jt/?vd_source=08295b5b4b3c5ece73fb91e3a54d202a) -## Build -```bash -docker build -t mnlm . -``` - -## Development -You can run the docker on the background and login to the docker container to do the development. - -0. Install the [Remote - Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) extension in VSCode. -1. Start the docker container: -```bash -docker-compose up -d -``` -2. In VSCode, click the "Open the remote window" button on the bottom left corner, and select "Remote-Containers: Attach to Running Container..." -3. Select the container you just started, should be something like `mnln-ros_dev_env-1`. And you will see a new VSCode window pop up. -4. Open the folder `/home/small-thinking/mnlm` in the new VSCode window, and you can start the development. +## Build and Start the Docker Container -### Start rviz2 -rviz2 is a visualization tool for ROS2. You can use it to visualize the robots. -Open a terminal in the docker container, and run the following command: +1. Install [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/). ```bash -root@809c4c72ba40:~# ros2 run rviz2 rviz2 - -QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root' -[INFO] [1703722219.404846429] [rviz2]: Stereo is NOT SUPPORTED -[INFO] [1703722219.404925804] [rviz2]: OpenGl version: 4.5 (GLSL 4.5) -[INFO] [1703722219.659990970] [rviz2]: Stereo is NOT SUPPORTED +docker-compose up --build -d ``` -Then in your host machine, open a browser and go to `http://localhost:8080/`. You will see the rviz2 file list. -![IMAGE ALT TEXT HERE](./images/rviz2-files.png "rviz2") +This command will build the docker image for the server side, and also start the docker container. +Users can also access the simulation environment through the browser by visiting `http://localhost:8080/vnc.html`. +[![IMAGE ALT TEXT HERE](./images/novnc.png)] -Click vnc.html, and then click the "Connect" button. You will see the rviz2 UI. - -### Start RQt -RQt is a graphical user interface framework that implements various tools and interfaces in the form of plugins. One can run all the existing GUI tools as dockable windows within RQt. - -Open a new terminal in the docker container, and run the following command: +2. Login to the docker container. +If you use vscode, you can install the `Dev Container` extension and open the running container. Otherwise, you can login to the docker container by running the following command: ```bash -root@809c4c72ba40:~/mnlm# rqt - -QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root' +docker exec -it mnln-ros_dev_env-1 /bin/bash ``` -You will see the RQt added to the rviz2 UI. -### Test with sample programs - -#### Test with demo_nodes_cpp -Run the demo_node_cpp talker: +3. Start the ROS2 simulation. ```bash -source /opt/ros/humble/setup.bash -ros2 run demo_nodes_cpp talker +cd /home/small-thinking/mnlm/mnlm/robot/robot_arm_ws ``` -In another terminal, run the demo_node_py listener: -```bash -source /opt/ros/humble/setup.bash -ros2 run demo_nodes_py listener ``` -#### Test with turtlesim -Run turtlesim_node: -```bash -root@809c4c72ba40:~/mnlm# ros2 run turtlesim turtlesim_node -QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root' -[INFO] [1703722372.104943222] [turtlesim]: Starting turtlesim with node name /turtlesim -[INFO] [1703722372.108501763] [turtlesim]: Spawning turtle [turtle1] at x=[5.544445], y=[5.544445], theta=[0.000000] +colcon build --symlink-install ; source install/setup.bash ; ros2 launch robot_arm robot_arm.launch.py ``` -In another terminal, run turtle_teleop_key: -```bash -root@809c4c72ba40:~/mnlm# ros2 run turtlesim turtle_teleop_key +You should see the server side program started. And you can visit `http://localhost:8080/vnc.html`` to see the Gazebo Fortress simulation environment. +![IMAGE ALT TEXT HERE](./images/gazebo.png) -Reading from keyboard ---------------------------- -Use arrow keys to move the turtle. -Use G|B|V|C|D|E|R|T keys to rotate to absolute orientations. 'F' to cancel a rotation. -'Q' to quit. -``` -![IMAGE ALT TEXT HERE](./images/turtlesim.png "turtlesim") +4. In your host machine. You can run the command to start the voice based UI. +![IMAGE ALT TEXT HERE](./images/voice.png) -## Cleanup untagged images ```bash -docker rm $(docker ps -a -q) ; docker images | grep '' | awk '{print $3}' | xargs docker rmi -``` - - +In the project folder: +```bash +cd mnlm/client/gpt_control +``` +Then you can see the client side started, and you will be prompoted to tell the robot what to do. -TODOs: -5. Build a key value store to store the verbal command and the list of commands. -6. Index the key value store with the verbal command in to vector db. -7. Add RAG after voice recognition. \ No newline at end of file +```bash +python assistant.py +``` \ No newline at end of file diff --git a/developer.md b/developer.md new file mode 100644 index 0000000..514d2fe --- /dev/null +++ b/developer.md @@ -0,0 +1,31 @@ + + +# Development + +## Project structure +``` + +mnlm +|-> mnlm +| |--> client +| |--> gpt_control +| |--> assistant.py # voice based UI start program +| |--> command_indexer.py # script to index commands into the knowledge base +| |--> knowledge # the knowledge base +| |--> dummy_robot_arm_server.py # dummy server for testing purpose so we don't need to start ROS2 server +| |--> robot +| |--> robot_arm_ws # ROS2 workspace +| |--> robot_arm # ROS2 package +| |--> launch # launch file +| |--> config # configuration file, includes the ros2 control configuration +| |--> src # source code +| |--> models # robot model, includes the xacro and sdf files +| |--> setup.py # setup file for the package +| |--> package.xml # package file +|-> docker-compose.yml # docker compose file +``` + +## Cleanup untagged images +```bash +docker rm $(docker ps -a -q) ; docker images | grep '' | awk '{print $3}' | xargs docker rmi +``` diff --git a/images/gazebo.png b/images/gazebo.png new file mode 100644 index 0000000..a126ce5 Binary files /dev/null and b/images/gazebo.png differ diff --git a/images/noVNC.png b/images/noVNC.png new file mode 100644 index 0000000..12743e8 Binary files /dev/null and b/images/noVNC.png differ diff --git a/images/voice.png b/images/voice.png new file mode 100644 index 0000000..8c71ba5 Binary files /dev/null and b/images/voice.png differ diff --git a/mnlm/client/knowledge/command_bank.json b/mnlm/client/knowledge/command_bank.json index 9084f89..31400b4 100644 --- a/mnlm/client/knowledge/command_bank.json +++ b/mnlm/client/knowledge/command_bank.json @@ -63,15 +63,47 @@ }, { "operation": "move_single_servo", - "parameters": {"id": "servo3", "angle": 10, "time": 500} + "parameters": {"id": "servo3", "angle": 0, "time": 500} }, { - "operation": "move_single_servo", - "parameters": {"id": "servo3", "angle": 50, "time": 500} + "operation": "move_all_servos", + "parameters": {"angles": [0, 0, 0, 0, 0, 0, 0], "time": 500} + } + ] + }, + "shake the body": { + "operations": [ + { + "operation": "move_all_servos", + "parameters": {"angles": [45, 60, 60, 30, 0, 0, 0], "time": 500} }, { - "operation": "move_single_servo", - "parameters": {"id": "servo3", "angle": 0, "time": 500} + "operation": "move_all_servos", + "parameters": {"angles": [0, 30, 30, 0, 0, 0, 0], "time": 500} + }, + { + "operation": "move_all_servos", + "parameters": {"angles": [135, 60, 60, 30, 0, 0, 0], "time": 500} + }, + { + "operation": "move_all_servos", + "parameters": {"angles": [0, 0, 0, 0, 0, 0, 0], "time": 500} + } + ] + }, + "seize the gripper, size the fingers": { + "operations": [ + { + "operation": "move_all_servos", + "parameters": {"angles": [0, 0, 0, 0, 0, 10, 10], "time": 500} + }, + { + "operation": "move_all_servos", + "parameters": {"angles": [0, 0, 0, 0, 0, 0, 0], "time": 500} + }, + { + "operation": "move_all_servos", + "parameters": {"angles": [0, 0, 0, 0, 0, 10, 10], "time": 500} }, { "operation": "move_all_servos", diff --git a/mnlm/client/knowledge/index/instructions.index b/mnlm/client/knowledge/index/instructions.index index ba4653d..67bf252 100644 Binary files a/mnlm/client/knowledge/index/instructions.index and b/mnlm/client/knowledge/index/instructions.index differ diff --git a/mnlm/client/knowledge/index/instructions_data.json b/mnlm/client/knowledge/index/instructions_data.json index 12d9073..75292c8 100644 --- a/mnlm/client/knowledge/index/instructions_data.json +++ b/mnlm/client/knowledge/index/instructions_data.json @@ -152,23 +152,137 @@ "operation": "move_single_servo", "parameters": { "id": "servo3", - "angle": 10, + "angle": 0, "time": 500 } }, { - "operation": "move_single_servo", + "operation": "move_all_servos", "parameters": { - "id": "servo3", - "angle": 50, + "angles": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "time": 500 + } + } + ] + }, + { + "instruction": "shake the body", + "operations": [ + { + "operation": "move_all_servos", + "parameters": { + "angles": [ + 45, + 60, + 60, + 30, + 0, + 0, + 0 + ], "time": 500 } }, { - "operation": "move_single_servo", + "operation": "move_all_servos", "parameters": { - "id": "servo3", - "angle": 0, + "angles": [ + 0, + 30, + 30, + 0, + 0, + 0, + 0 + ], + "time": 500 + } + }, + { + "operation": "move_all_servos", + "parameters": { + "angles": [ + 135, + 60, + 60, + 30, + 0, + 0, + 0 + ], + "time": 500 + } + }, + { + "operation": "move_all_servos", + "parameters": { + "angles": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "time": 500 + } + } + ] + }, + { + "instruction": "seize the gripper, size the fingers", + "operations": [ + { + "operation": "move_all_servos", + "parameters": { + "angles": [ + 0, + 0, + 0, + 0, + 0, + 10, + 10 + ], + "time": 500 + } + }, + { + "operation": "move_all_servos", + "parameters": { + "angles": [ + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "time": 500 + } + }, + { + "operation": "move_all_servos", + "parameters": { + "angles": [ + 0, + 0, + 0, + 0, + 0, + 10, + 10 + ], "time": 500 } }, diff --git a/mnlm/robot/robot_arm_ws/src/robot_arm/robot_arm/command_dispatcher_node.py b/mnlm/robot/robot_arm_ws/src/robot_arm/robot_arm/command_dispatcher_node.py index 881ce1f..4ef33c7 100644 --- a/mnlm/robot/robot_arm_ws/src/robot_arm/robot_arm/command_dispatcher_node.py +++ b/mnlm/robot/robot_arm_ws/src/robot_arm/robot_arm/command_dispatcher_node.py @@ -71,7 +71,7 @@ def _process_command(self, operations: List[Any]): else: self.get_logger().error(f"Unknown action: {operation}") seconds = 5 - self._log_with_color("Waiting for {seconds} seconds before sending the next operation.") + self._log_with_color("Waiting for {seconds} seconds before sending the next operation.", "yellow") time.sleep(seconds) diff --git a/mnlm/robot/robot_arm_ws/src/robot_arm/rviz/view_robot.rviz b/mnlm/robot/robot_arm_ws/src/robot_arm/rviz/view_robot.rviz deleted file mode 100644 index 709838f..0000000 --- a/mnlm/robot/robot_arm_ws/src/robot_arm/rviz/view_robot.rviz +++ /dev/null @@ -1,198 +0,0 @@ -Panels: - - Class: rviz_common/Displays - Help Height: 78 - Name: Displays - Property Tree Widget: - Expanded: - - /RobotModel1 - Splitter Ratio: 0.5 - Tree Height: 766 - - Class: rviz_common/Selection - Name: Selection - - Class: rviz_common/Tool Properties - Expanded: - - /2D Goal Pose1 - - /Publish Point1 - Name: Tool Properties - Splitter Ratio: 0.5886790156364441 - - Class: rviz_common/Views - Expanded: - - /Current View1 - Name: Views - Splitter Ratio: 0.5 -Visualization Manager: - Class: "" - Displays: - - Alpha: 0.5 - Cell Size: 1 - Class: rviz_default_plugins/Grid - Color: 160; 160; 164 - Enabled: true - Line Style: - Line Width: 0.029999999329447746 - Value: Lines - Name: Grid - Normal Cell Count: 0 - Offset: - X: 0 - Y: 0 - Z: 0 - Plane: XY - Plane Cell Count: 10 - Reference Frame: - Value: true - - Class: rviz_default_plugins/TF - Enabled: true - Frame Timeout: 15 - Frames: - All Enabled: true - arm_link: - Value: true - base_link: - Value: true - camera_link: - Value: true - slider_link: - Value: true - world: - Value: true - Marker Scale: 1 - Name: TF - Show Arrows: true - Show Axes: true - Show Names: true - Tree: - world: - base_link: - slider_link: - arm_link: - camera_link: - {} - Update Interval: 0 - Value: true - - Alpha: 0.800000011920929 - Class: rviz_default_plugins/RobotModel - Collision Enabled: false - Description File: "" - Description Source: Topic - Description Topic: - Depth: 5 - Durability Policy: Volatile - History Policy: Keep Last - Reliability Policy: Reliable - Value: /robot_description - Enabled: true - Links: - All Links Enabled: true - Expand Joint Details: false - Expand Link Details: false - Expand Tree: false - Link Tree Style: Links in Alphabetic Order - arm_link: - Alpha: 1 - Show Axes: false - Show Trail: false - Value: true - base_link: - Alpha: 1 - Show Axes: false - Show Trail: false - Value: true - camera_link: - Alpha: 1 - Show Axes: false - Show Trail: false - Value: true - slider_link: - Alpha: 1 - Show Axes: false - Show Trail: false - Value: true - world: - Alpha: 1 - Show Axes: false - Show Trail: false - Name: RobotModel - TF Prefix: "" - Update Interval: 0 - Value: true - Visual Enabled: true - Enabled: true - Global Options: - Background Color: 48; 48; 48 - Fixed Frame: world - Frame Rate: 30 - Name: root - Tools: - - Class: rviz_default_plugins/Interact - Hide Inactive Objects: true - - Class: rviz_default_plugins/MoveCamera - - Class: rviz_default_plugins/Select - - Class: rviz_default_plugins/FocusCamera - - Class: rviz_default_plugins/Measure - Line color: 128; 128; 0 - - Class: rviz_default_plugins/SetInitialPose - Topic: - Depth: 5 - Durability Policy: Volatile - History Policy: Keep Last - Reliability Policy: Reliable - Value: /initialpose - - Class: rviz_default_plugins/SetGoal - Topic: - Depth: 5 - Durability Policy: Volatile - History Policy: Keep Last - Reliability Policy: Reliable - Value: /goal_pose - - Class: rviz_default_plugins/PublishPoint - Single click: true - Topic: - Depth: 5 - Durability Policy: Volatile - History Policy: Keep Last - Reliability Policy: Reliable - Value: /clicked_point - Transformation: - Current: - Class: rviz_default_plugins/TF - Value: true - Views: - Current: - Class: rviz_default_plugins/Orbit - Distance: 3.431431531906128 - Enable Stereo Rendering: - Stereo Eye Separation: 0.05999999865889549 - Stereo Focal Distance: 1 - Swap Stereo Eyes: false - Value: false - Focal Point: - X: 1.8107112646102905 - Y: 0.7177653312683105 - Z: 0.437046080827713 - Focal Shape Fixed Size: true - Focal Shape Size: 0.05000000074505806 - Invert Z Axis: false - Name: Current View - Near Clip Distance: 0.009999999776482582 - Pitch: 0.4753981828689575 - Target Frame: - Value: Orbit (rviz) - Yaw: 5.143584728240967 - Saved: ~ -Window Geometry: - Displays: - collapsed: false - Height: 989 - Hide Left Dock: false - Hide Right Dock: true - QMainWindow State: 000000ff00000000fd00000004000000000000015600000387fc0200000008fb0000001200530065006c0065006300740069006f006e00000001e10000009b0000005c00fffffffb0000001e0054006f006f006c002000500072006f007000650072007400690065007302000001ed000001df00000185000000a3fb000000120056006900650077007300200054006f006f02000001df000002110000018500000122fb000000200054006f006f006c002000500072006f0070006500720074006900650073003203000002880000011d000002210000017afb000000100044006900730070006c006100790073010000003b00000387000000c700fffffffb0000002000730065006c0065006300740069006f006e00200062007500660066006500720200000138000000aa0000023a00000294fb00000014005700690064006500530074006500720065006f02000000e6000000d2000003ee0000030bfb0000000c004b0069006e0065006300740200000186000001060000030c00000261000000010000010f00000387fc0200000003fb0000001e0054006f006f006c002000500072006f00700065007200740069006500730100000041000000780000000000000000fb0000000a00560069006500770073000000003b00000387000000a000fffffffb0000001200530065006c0065006300740069006f006e010000025a000000b200000000000000000000000200000490000000a9fc0100000001fb0000000a00560069006500770073030000004e00000080000002e10000019700000003000004420000003efc0100000002fb0000000800540069006d00650100000000000004420000000000000000fb0000000800540069006d00650100000000000004500000000000000000000005340000038700000004000000040000000800000008fc0000000100000002000000010000000a0054006f006f006c00730100000000ffffffff0000000000000000 - Selection: - collapsed: false - Tool Properties: - collapsed: false - Views: - collapsed: true - Width: 1680 - X: 0 - Y: 0 \ No newline at end of file