The PokéTechs - a project by Jenny, Lee, Nicole, Austin, and William!

March 2021, for LA Hacks

Hello!

We wanted to build a project to learn more about machine learning and app development. It was intimidating without prior experience, but in the end we successfully trained a model that classifies what animal is present in a picture, and built an app around it inspired by a Pokédex.

• note: the model used to classify images for the app requires authentification and requires being deployed, so you won't be able to try it currently! *

Inspiration

• Pokémon!!
• Computer Vision
• iNaturalist concept and dataset

What it does

• Uses a cloud trained classification model to identify the animal from an image
• Generates a Pokémon card for "captured" specimen

How we built it

• Extracted subset of iNaturalist images dataset that was of interest (more common animals)
• Generated CSV for dataset annotation (multi-label) using Python
• Trained custom model from our dataset using Google Vision AutoML
• Built the app using React-Native (Expo)
• Designed elements with Figma

Challenges we ran into

• Dealing with such a large dataset that initially had thousands of different classification labels
• Learning React-Native, especially how to properly make requests to our custom model in the cloud

Accomplishments that we're proud of

• Having a trained model that can make somewhat accurate predictions for common animals
• Having a fun and nostalgic interface!
• Creating an app where you can navigate across different screens while preserving information

What we learned

• How to use Google Cloud and Vision AutoML
• What goes into app creation
• React-Native: navigation, styling, camera permissions and usage

What's next for Poke-techs

• Use Firebase to set up accounts so users can save and share "captured" animals
• Set up location so users have a checklist of all local animals in order to "catch them all!"
• Use iNaturalist annotations and/or web-scraping to get facts about the animal identified to display to user
• Improve our custom ML model to recognize differences between more similar species with better recall and precision
• Move ML model offline for local download so money/credits is not wasted through continual deployment
• Improve styling with gifs, custom animations across screens, and elements which more closely emulate Pokémon designs