Research in Autonomous AI Driven Robotics

See our project write-up and demo videos here

This repository is authored by Jesse Gilbert and Liam Campbell for the Center for Engineering & Education Outreach (CEEO) at Tufts University.

Introduction

This project aims to explore the role that AI can play in coding robotics in two parts…

Program for AI Robotics Engineering (PARE) - This repo:

Focuses on the core capabilities of AI-programmed robotics, such as autonomous decision making, direct hardware control, iterative coding, and independent verification of success.

Scaffolded Program for AI Robotics Engineering (SPARE) - Repo here:

Builds on the foundation of PARE but introduces a network of manager and worker AI Assistants. This hierarchical system experiments with the execution of more complex tasks, that address multiple microcontrollers, by delegating them to sub-workers and abstracting them from the user.

Usage

To use this repository, follow these steps:

• Ensure you have Python installed on your system.
• Clone this repository to your local machine.
• Install all required dependencies:
  • pip install openai (for calling the OpenAI API)
  • pip install opencv-python (for capturing images for the get_visual_feedback function)
  • pip install pillow (for opening images for the get_visual_feedback function)
  • pip install serial (for communicating with the robot of USB)
• Create an OpenAI API key here.
• Set up your API key by following the instructions here.
• Plug in your LEGO SPIKE Prime to your computer and turn it on.
• Find the serial port of your LEGO SPIKE Prime device and update it in main.py.
• Run main.py to start the interactive AI powered coding session.

File Descriptions

main.py: Orchestrates the interaction between the user and the AI assistant in the AiAlchemy class.

ai_alchemy.py: Contains the class AIAlchemy, which provides the framework for OpenAI's AI model to interface with the user, the robot, documentation, etc.

serial_interface.py: Handles all serial communication between the computer and LEGO SPIKE Prime.

query_dict.json: Contains necessary syntax and documentation for MicroPython coding on the LEGO SPIKE Prime platform. Ultimately we hope that this JSON file will be replaced by documentation built into the microcontroller itself. For now, we are using this isolated JSON documentation file to mimic the idea that the AI model has no pre-established knowledge of the robotic platform it is programming.