Skip to content

Commit

Permalink
refactor: Refactor project structure and remove ml folder (#30)
Browse files Browse the repository at this point in the history
  • Loading branch information
nicolantean authored Oct 1, 2024
1 parent dbc9001 commit 72ad5b1
Show file tree
Hide file tree
Showing 621 changed files with 219 additions and 15,295 deletions.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ jobs:
bundler-cache: true
- name: Activate arb_utils
run: |
dart pub global activate arb_utils 0.8.1
dart pub global activate arb_utils 0.8.3
- name: Get dependencies
uses: maierj/[email protected]
with:
Expand Down
File renamed without changes.
54 changes: 52 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,66 @@
*.swp
.DS_Store
.atom/
build/
.buildlog/
.history
.svn/
migrate_working_dir/

# Mason
.mason/
mason-lock.json

# IntelliJ related
*.iml
*.ipr
*.iws
.idea/*
android/.idea
!.idea/runConfigurations/

# The .vscode folder contains launch configuration and tasks you configure in
# VS Code which you may wish to be included in version control, so this line
# is commented out by default.
.vscode/settings.json

# Flutter/Dart/Pub related
**/doc/api/
**/ios/Flutter/.last_build_id
.dart_tool/
.flutter-plugins
.flutter-plugins-dependencies
.packages
.pub-cache/
.pub/
/build/

# Web related
lib/generated_plugin_registrant.dart

# Symbolication related
app.*.symbols

# Obfuscation related
app.*.map.json

# Android Studio will place build artifacts here
/android/app/debug
/android/app/profile
/android/app/release

# Environment files
*.env
*.env.default

# Fvm
.fvm/

# fastlane specific
**/fastlane/report.xml
**/fastlane/Preview.html
**/fastlane/screenshots
**/fastlane/test_output
.bundle/
vendor/bundle/

#intl
l10n_errors
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
167 changes: 165 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,167 @@
# XL Says
# SimonSays AI

This is the readme for XL Says. /ml folder contains the ML model implementation
The **SimonSays AI Game** is a modern twist on the classic **Simon Says** game. In the original Simon Says, players must follow a sequence of instructions, like "Simon says touch your nose," but only if the command is prefixed with "Simon says". If players perform an action without the prefix, they lose. Another popular variant uses colors and sound sequences, where players replicate increasingly complex color patterns displayed by the game.

#### **How This Version is Different:**

In our version, instead of colors or spoken commands, the game uses **hand gestures** that players must replicate using a camera. Here's how it works:

- **Gestures as Commands:** Instead of colored buttons, the game displays a sequence of hand gestures represented as emojis.
- **Player Input:** The player must replicate the sequence of gestures in the correct order using their hand in front of the camera.
- **On device Recognition:** The game uses machine learning models to recognize and evaluate the player’s gestures on device in real time.
- **Progression:** Just like in the original Simon Says, each round adds a new gesture to the sequence, making it progressively more challenging.

This modern version retains the core challenge of memory and coordination from the original **Simon Says**, but adds a layer of interactivity with **gesture-based input**, creating a more immersive and engaging gameplay experience.

<div align="center">
<img src="docs/assets/game_demo.gif" alt="Game demo">
<p><em>Figure 1: A small demo of the SimonSays AI game showcasing the hand gesture recognition and gameplay mechanics.</em></p>
</div>

## Project Overview

This project is based on the [Xmartlabs Flutter Template][flutter_template_link]. You can check it out for more details and to understand the foundational structure of this project.

## Solving the Gesture Recognition Problem

When developing the **Simon Says Game**, we explored two main approaches to solve the problem of on-device real-time hand gesture recognition:

1. [Direct Integration with TensorFlow Lite:](#integrating-tflite-models-with-flutter-plugin) This method involves using multiple TensorFlow Lite models within Flutter via the official **tflite plugin**. This approach provides flexibility, allowing us to run custom ML models directly within Flutter. However, it presented challenges related to performance, particularly with image preprocessing tasks.

2. [MediaPipe Integration:](#mediapipe-integration) The alternative approach leverages **MediaPipe**, a framework known for its efficiency in real-time ML processing, particularly for gesture recognition. While MediaPipe offers optimized performance, it lacks a native Flutter integration at the moment, requiring us to use native implementations and method channels for communication between Flutter and platform-specific code.

The following sections will outline how we implemented both approaches, the challenges faced, and the performance comparisons between them.

## **Integrating TFLite models with Flutter plugin**

The [Gesture Classifier][gesture_classifier] is responsible for real-time hand gesture recognition in the Simon Says game, using four [TensorFlow Lite][tf_lite] models integrated using the [TFLite Flutter][tflite_flutter] library. Each model performs a specific task, with the output of one model serving as the input for the next, ensuring accurate sequential processing.

<div align="center">
<img src="docs/assets/model_sequence.png" alt="Model sequence">
<p><em>Figure 2: Sequence of models used in the Gesture Classifier.</em></p>
</div>

- **Model 1: Hand Bounding Box Detection**

- Detects the hand's bounding box from the camera feed.
- Focuses on the area where the hand is located, cropping the image to improve subsequent model performance.

- **Model 2: Hand Landmarks Detection**

- Processes the cropped hand image from **Model 1**.
- Detects 21 key landmarks (joints and finger positions) on the hand, providing a detailed structure for gesture recognition.

- **Model 3: Landmarks to Embedding**

- Analyzes the landmarks detected by **Model 2**.
- Returns the embedding corresponding to the landmarks received as input.

- **Model 4: Gesture recognizer**

- Evaluates the embedding from **Model 3** in the context of the models.
- Returns the gesture label corresponding to the gesture recognized from the image given at the begining in the **Model 1** which can be one of these: unrecognized, closed, open, pointing_up, thumbs_down, thumbs_up, victory, love.

## **MediaPipe Integration**

Apart from integrating directly **TFLite models** for gesture recognition we also use [MediaPipe][mediapipe] as an alternative approach. **MediaPipe** is a powerful framework for building multimodal machine learning pipelines, developed by Google. It provides pre-trained models for hand gesture recognition and other ML tasks, offering high performance for real-time applications.

### **Why Use MediaPipe?**

While **MediaPipe** offers robust gesture recognition models, the current lack of a dedicated Flutter library for fully integrating its solutions required us to implement it using **native code**. This allowed us to take advantage of MediaPipe’s performance, but the integration process was more complex.

### **Integration with Flutter Using Method Channels**

To integrate **MediaPipe** with our Flutter application, we utilized **method channels**, which allow communication between the Flutter app (Dart code) and platform-specific native code (Android and iOS). Here’s how we approached it:

- **Native Implementation:** MediaPipe was set up natively on both Android and iOS platforms using Kotlin for Android and Swift for iOS.
- **Method Channels:** We used Flutter’s method channels to send and receive data between the native MediaPipe code and the Flutter app. This allowed us to pass the camera feed to the native side, where MediaPipe processed the gestures.
- **Gesture Recognition:** Once the gestures were processed by MediaPipe, the results were sent back to Flutter via method channels, where they were used to update the game logic in real-time.

### **Limitations and Benefits**

- **Benefit:** MediaPipe provides highly efficient gesture recognition and improved performance over the previous solution.
- **Limitation:** The integration is limited to using MediaPipe’s pre-trained models, and currently, there’s no direct Flutter library for easily customizing models.

## **Isolate-Based Model Execution**

To maintain a responsive UI and prevent blocking the main thread, the **Gesture Classifier** runs all TensorFlow Lite models on a separate **isolate**. In Flutter, isolates allow tasks to run in parallel, ensuring that computationally heavy operations, such as gesture recognition, do not interfere with the app's UI performance.

## **Game Manager**

The [Game Manager][game_manager] handles the core logic of the game, evaluating user gestures to determine if they match the expected sequence. It interacts with the gesture recognition pipeline and ensures smooth gameplay.

### **Key Responsibilities:**

- **Gesture Handling:**

- Receives classified gestures from the recognition models running in an isolate.
- Compares each incoming gesture to the current expected gesture in the sequence, updating the game state accordingly.

- **Sequence Matching:**

- Maintains an index of the correct gesture sequence.
- If a gesture matches the expected one, the game advances to the next gesture; if it doesn't, the round ends or resets.

- **Gesture Validation Window:**
- Implements a **400ms delay** between gesture evaluations to avoid false positives or negatives caused by incomplete transitions or accidental inputs.

---

Made with ❤️ by [Xmartlabs][xmartlabs].

## Contribute

👉 If you want to contribute please feel free to submit pull requests.

👉 If you have a feature request please [open an issue][open_issue].

👉 If you found a bug or [need help][need_help] please let us know.

👉 If you enjoy using Fluttips we would love to hear about it! Drop us a line on [X][xmartlabs_x].

## License

```
Copyright (c) 2024 Xmartlabs SRL
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
```

[design_system]: https://github.com/xmartlabs/simon-ai/tree/main/design_system
[app_router]: https://github.com/xmartlabs/simon-ai/blob/main/lib/ui/app_router.dart
[bloc]: https://bloclibrary.dev
[auto_route]: https://pub.dev/packages/auto_route
[models]: https://github.com/xmartlabs/simon-ai/tree/main/lib/core/model
[repository_folder]: https://github.com/xmartlabs/simon-ai/tree/main/lib/core/repository
[data_source_folder]: https://github.com/xmartlabs/simon-ai/tree/main/lib/core/source
[get_it]: https://pub.dev/packages/get_it
[scripts]: https://github.com/xmartlabs/simon-ai/tree/main/scripts
[flutter_template_link]: https://github.com/xmartlabs/flutter-template
[game_manager]: https://github.com/xmartlabs/simon-ai/tree/main/lib/core/repository/game_manager.dart
[gesture_classifier]: https://github.com/xmartlabs/simon-ai/tree/main/lib/core/hand_models/hand_gesture_classifier/hand_classifier.dart
[mediapipe]: https://ai.google.dev/edge/mediapipe/solutions/guide
[isolates]: https://docs.flutter.dev/perf/isolates
[tf_lite]: https://ai.google.dev/edge/litert
[xmartlabs]: https://xmartlabs.com/
[open_issue]: https://github.com/xmartlabs/simon-ai/issues/new
[need_help]: https://github.com/xmartlabs/simon-ai/issues/new
[xmartlabs_x]: https://x.com/xmartlabs
[tflite_flutter]: https://pub.dev/packages/tflite_flutter
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes.
File renamed without changes.
File renamed without changes.
8 changes: 0 additions & 8 deletions bitrise.yml
Original file line number Diff line number Diff line change
Expand Up @@ -129,10 +129,6 @@ workflows:
inputs:
- clone_depth: "15"
- cache-pull@2: {}
- change-workdir@1:
inputs:
- is_create_path: "false"
- path: client
- script@1:
inputs:
- script_file_path: null
Expand Down Expand Up @@ -224,7 +220,3 @@ app:
{
}
trigger_map:
- pull_request_target_branch: main
type: pull_request
workflow: run_test_and_analyzer
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
55 changes: 0 additions & 55 deletions client/.github/pull_request_template.md

This file was deleted.

70 changes: 0 additions & 70 deletions client/.gitignore

This file was deleted.

Loading

0 comments on commit 72ad5b1

Please sign in to comment.