Object detection using TensorFlow Lite on Android

Overview

This is an android app for object detection using TensorFlow Lite on a mobile device. I have used a pretrained MobileNet SSD quantized model from here. It is trained on the ImageNet dataset which can detect about 90 classes including banana, scissors, laptop, remote, vase etc. The main issue faced earlier of object detection on mobile is that the models are often too big to run. Tensorflow Lite solves this problem by providing a lightweight solution to run machine learning models on mobile. You can see here yourself how nicely the app works and detects multiple objects quickly.

TensorFlow Lite Model

TensorFlow Lite is not designed to train a model, the model can be trained on a higher power device. Then, the pretrained model can be converted to a TensorFlow Lite format (.tflite), which has a smaller footprint than can be easily run on a mobile or other embedded devices for classification, regresion or other such tasks. The model (.tflite) file and the class labels (.txt) file need to be placed in the assets folder of the android app.

The Android App for Object Detection

I have followed the TensorFlow Lite example for Object Detection. In this app we will get a running feed from the mobile device camera, then, run object detection on the frame in background, and then overlay the results of object detection on the frame with a bounding box.

First step here is to create an android app using Android Studio. My main Activity is MainActivity which will invoke the object detector. It extends the CameraActivity which in turn uses a CameraConnectionFragment to manage all camera related stuff.

The object detector is encapsulated by MobileNetObjDetector which uses the TensorFlow Lite Interpreter.

import org.tensorflow.lite.Interpreter;

Its very easy to initialize the Interpreter with the model:

private Interpreter tflite; tfLite = new Interpreter(loadModelFile(assetManager));

For object detection, it feeds an image of size 300x300 to the model and obtains the output as defined by the model.

tfLite.runForMultipleInputsOutputs(inputArray, outputMap);

Then the MobileNetObjDetector convertes the outputMap into a List of DetectionResult which can be easily consumed for painting the overlay. Each DetectionResult has the label detected, the confidence score of the detection and the bounding box of the detection.

The OverlayView takes care of resizing the bounding bozes as per the mobile device screen preview size and render it on top of the camera frame.

Results

Dining Table with Cups

TV

Microwave

Checkout more in a video here