RPi-camera-glasses

This project was prompted by GPT-4's visual input feature, which allows image upload and analysis: https://openai.com/product/gpt-4. Since that feature has not been released as of 7/18/2023, I will be using a custom Google Bard API by Daniel Park: https://github.com/dsdanielpark/Bard-API. I will be attaching a Raspberry Pi camera module onto my glasses, capturing images upon pressing a button wired to the Pi, and sending those images to https://bard.google.com/.

Be mindful that Google may rate limit you if you make excessive requests to the website.

Results

Bard's description is almost perfect, the only error being the "T" misidentified as "Y"

Bard's description is mostly correct, including correctly identifying the majority of the text present. Color assessment is inaccurate

Future Changes

• Implement a USB microphone to create custom prompts for each image (input and output audio together is not simple with Raspberry Pi)
• Implement a more stable API (this will require Google or OpenAI to add image analysis with LLMs to their APIs)
• Make camera glasses more compact and useable in real-world scenarios

Materials

• Raspberry Pi 4 Model B
• Raspberry Pi Camera Module V3
• Miuzei case
• Momentary push button (I recommend getting a bigger one than this)
• Arducam camera module extension cable
• Arducam camera extension cable
• Dupont wires
• Jonkuu Power Bank
• Soldering iron (depending on type of push button)

Installation

1. Assemble the hardware. The camera extension cable should be wired from camera to camera module, the camera module should be attached to the Pi's CSI port, and the push button should be wired to any GPIO pin and ground.
2. Install the recommended Raspberry Pi OS on a microSD card: https://www.raspberrypi.com/software/
3. Boot up Raspberry Pi and open a CLI
4. Clone repository: git clone https://github.com/senseibence/RPi-camera-glasses.git
5. Change directory: cd RPi-camera-glasses
6. Install packages:

• All packages except for 3 (openai (unused), bardapi, and gTTS) are included in the recommended Raspberry Pi OS (Bullseye). If you still wish to install them manually, run pip install -r requirements.txt. Otherwise, install just the 2 used packages (bardapi and gTTS). Using a virtual environment is recommended; however, I encountered issues with installing picamera2 because of PyQt5
• Run sudo apt install mpg123 to install an MP3 player onto your system

7. Set the value of Bard_Cookie_Value in keys.json:

• Go to https://bard.google.com/
• F12 for developer console
• Application → Cookies → Copy the value of __Secure-1PSID cookie.

8. Run python main.py to start the program

Hardware Images

Rough diagram of the project

Raspberry Pi 4 Model B before hardware installations

Raspberry Pi 4 Model B after hardware installations

Glasses with camera and module