Adaptive learning

README.md

@@ -182,6 +182,23 @@ docker build -f x86-openvino.Dockerfile -t "neuralet/smart-social-distancing:lat
 docker run -it -p HOST_PORT:8000 -v "$PWD/data":/repo/data -v "$PWD/config-x86-openvino.ini":/repo/config-x86-openvino.ini -v "$PWD/certificates/processor":/repo/certs  -e TZ=`./timezone.sh` neuralet/smart-social-distancing:latest-x86_64_openvino
 ```
 
+##### Run on AMD node with a connected Coral USB Accelerator with Adaptive Learning Support
+Currently Smart Social Distancing supports Adaptive Learning framework on X86 nodes with connected USB TPU. for more information about Adaptive Learning visit [this page](https://github.com/neuralet/neuralet/tree/master/applications/adaptive-learning)
+Important notes:
+1. you must install [Docker Compose](https://github.com/docker/compose) to be able to use Adaptive Learning.
+2. At this time Adaptive Learning is only compatible with `mobilenet_ssd_v2` models.
+
+```bash
+# Download model first:
+./download_x86_model.sh
+
+# 1) Configure adaptive-learing/configs/iterdet.ini file. (you may only need to configure video VideoPath, BatchSize and Epochs)
+# 2) Configure the HOST_PORT in docker-compose-adaptive-learning-usbtpu.yml
+# 3) Run the Docker Compose 
+docker-compose -f docker-compose-adaptive-learning-usbtpu.yml up --build
+```
+
+
 ### Configurations
 You can read and modify the configurations in `config-*.ini` files, accordingly:
 

adaptive-learning/README.md

@@ -0,0 +1,64 @@
+# Adaptive Learning  of Object Detection Models
+
+
+The goal of adaptive learning is to build robust systems that can work efficiently and adapt to new environments (novel data distributions) without labeled data or manual labeling or storing or transferring data from the edge devices to other systems. This framework will dramatically simplify machine learning systems’ architecture and lower their deployment and maintenance costs when applied.
+We published a comprehensive blog about the concept and features of Adaptive Learning. Please refer to the [blog post](https://neuralet.com/article/adaptive-learning-part-1/) for more details.
+
+## prerequisites
+1. At least one Nvidia GPU
+2. At least 6GB of RAM
+3. [Docker](https://docs.docker.com/get-docker/)
+4. [Docker Compose](https://github.com/docker/compose)
+5. [Nvidia Docker Toolkit](https://github.com/NVIDIA/nvidia-docker)
+
+## Getting Started
+### Clone Neuralet Repository
+Make sure you have the prerequisites and then clone this repository to your local system by running this command:
+```
+git clone https://github.com/neuralet/neuralet.git
+
+cd neuralet/applications/adaptive-learning
+```
+### Download Sample Video (In Case of Trying Demo):
+
+```
+bash ./download_sample_video.sh
+```
+### Configure Adaptive Learning Config File
+To customize the Adaptive Learning framework based on your needs, you must configure one of the sample config files on `configs/` directory.
+there is a brief explanation on each parameter of config files in the following table:
+
+
+| Parameter    | Options | Comments                                                                                                                          |
+| ------------ | ------- | -------- |
+| Teacher/Name     | iterdet     | name of the teacher model. (in case of implementation of new teacher you should register its name in `teachers/model_builder.py`)     |
+| Teacher/ImageSize | 300,300,3 (or any other appropriate image size) | the input image size of the teacher model |
+| Teacher/ClassID | integer | The pedestrian class ID of the Teacher. (this parameter is important only where the teacher model is a multiple class object detector.) |
+| Teacher/MinScore | a float number between 0 and 1 | the teacher model threshold for detecting an object |
+| Teacher/VideoPath | a unix-like path | path to the video file that you want to apply adaptive learning. (don't change it if you just want run demo)  |
+| Teacher/MaxAllowedImage | an integer number | Maximum number of images that can be stored in hard, when this threshold is exeeded, the teacher model stops processing video frames until the student models picks some of the images and remove them from hard disk. |
+| Teacher/MinDetectionPerFrame | an integer number | The teacher only stores the frame only if it detects at least this many objects, otherwise it discards the frame.|
+| Teacher/SaveFrequency | An integer bigger than 0 | The teacher model will store video frames and the corresponding predictions with this frequency |
+| Teacher/PostProcessing | one of `"background_filter"` or `" "` | Background filter will apply a background subtraction algorithm on video frames and discards the bounding boxes in which their background pixels rate is higher than a defined threshold. |
+| Teacher/ImageFeature | One of the `"foreground_mask"`, `"optical_flow_magnitude"`, `"foreground_mask && optical_flow_magnitude"` or `" "` |This parameter specifies the type of input feature engineering that will perform for training. `"foreground_mask"` replaces one of the RGB channels with the foreground mask. `"optical_flow_magnitude"` replaces one of the RGB channels with the magnitude of optical flow vectors and, `"foreground_mask && optical_flow_magnitude"` performs two feature engineering technique at the same time as well as changing the remaining RGB channel with the grayscale transformation of the frame. For more information about feature engineering and their impact on the model's accuracy, visit our blog.
+| Student/BatchSize | An integer bigger than 0 | The student's training batch size |
+| Student/Epochs | An integer bigger than 0 | The student's training number of epochs in each round |
+| Student/ValidationSplit | An float between 0 and 1 | the portions of data which will be used for validation in each training round |
+| Student/ExamplePerRound | An integer bigger than 0 | How many example to use for each training round? |
+| Student/TrainingRounds | An integer bigger than 0 | how many rounds you want to run Adaptive Learning? |
+| Student/QuantizationAware | true or false | whether to train the student model with quantization aware strategy or not. This is specially useful when you want to deploy the final model on a edge device that only supports `Int8` precision like Edge TPU. By applying quantization aware training the student model will be exported to `tflite` too.|
+
+
+### Running the Docker Compose
+
+After configuring the config file, you just need to run following command to start the Adaptive Learning:
+```
+docker-compose up
+```
+After the training rounds completed you can find the final student model under: `data/student_model/frozen_graph/`
+
+### Tracking Training Process
+In the `docker-compose.yml` file, you can forward the 6006 port (tensorboard port) of the student's container to one of your machine's open ports. then browse this port in the browser to track and monitor training with tensorboard.
+
+### Defining Your Own Teacher Model.
+You can add your own teacher model to the Adaptive Learning framework by writing a class which is a subclass of `TeacherMetaArcht` class and implement `inference` method. please check out the `teachers/teacher_meta_arch.py` and `teachers/iterdet.py` for more information.

adaptive-learning/configs/config_engine.py

@@ -0,0 +1,82 @@
+#!/usr/bin/python3
+import configparser
+import threading
+
+
+class ConfigEngine:
+    """
+    Handle the .ini confige file and provide a convenient interface to read the parameters from config.
+    When an instance of ConfigeEngine is created you can use/pass it to other classes/modules that needs
+    access to the parameters at config file.
+
+    :param config_path: the path of config file
+    """
+
+    def __init__(self, config_path='./config-skeleton.ini'):
+        self.config = configparser.ConfigParser()
+        self.config.optionxform = str
+        self.config_file_path = config_path
+        self.lock = threading.Lock()
+        # For dynamic and cross-chapter flexible parameters: 
+        self.config._interpolation = configparser.ExtendedInterpolation()
+        self.section_options_dict = {}
+        self._load()
+
+    def set_config_file(self, path):
+        self.lock.acquire()
+        try:
+            self.config.clear()
+            self.config_file_path = path
+            self._load()
+        finally:
+            self.lock.release()
+
+    def _load(self):
+        self.config.read(self.config_file_path)
+        for section in self.config.sections():
+            self.section_options_dict[section] = {}
+            options = self.config.options(section)
+            for option in options:
+                try:
+                    val = self.config.get(section, option)
+                    self.section_options_dict[section][option] = val
+                    if val == -1:
+                        print("skip: %s" % option)
+                except:
+                    print("exception on %s!" % option)
+                    self.section_options_dict[section][option] = None
+
+    def save(self, path):
+        self.lock.acquire()
+        try:
+            file_obj = open(path, "w")
+            self.config.write(file_obj)
+            file_obj.close()
+        finally:
+            self.lock.release()
+
+    def get_section_dict(self, section):
+        return self.section_options_dict[section]
+
+    def get_boolean(self, section, option):
+        result = None
+        self.lock.acquire()
+        try:
+            result = self.config.getboolean(section, option)
+        finally:
+            self.lock.release()
+
+    def toggle_boolean(self, section, option):
+        self.lock.acquire()
+        try:
+            val = self.config.getboolean(section, option)
+            self.config.set(section, option, str(not val))
+        finally:
+            self.lock.release()
+
+    def set_option_in_section(self, section, option, value):
+        self.lock.acquire()
+        try:
+            self.config.set(section, option, value)
+        finally:
+            self.lock.release()

adaptive-learning/configs/iterdet.ini

@@ -0,0 +1,31 @@
+[Teacher]
+Name: iterdet
+;ImageSize should be 3 numbers seperated by commas, no spaces: 300,300,3
+ImageSize: 704,576,3
+;Pedestrian class ID of the teacher detector
+ClassID: 1
+;Score threshold of detector
+MinScore: 0.5
+VideoPath: /repo/data/softbio_vid.mp4
+;Directory that stores the video frames and annotations
+DataDir: /repo/adaptive-learning/data
+ImagePath: /repo/adaptive-learning/data/images
+XmlPath: /repo/adaptive-learning/data/xmls
+;Maximum number of images that can be stored in hard simultaneously
+MaxAllowedImage: 10000
+;Minimum number of instance in each frame to store the frame
+MinDetectionPerFrame: 1
+;frequency of saving frames
+SaveFrequency: 1
+Device: GPU
+PostProcessing: " "
+ImageFeature: " "
+[Student]
+BatchSize: 4
+Epochs: 20
+ValidationSplit: .17
+;how many data picks for training in each round
+ExamplePerRound: 2000
+TrainingRounds: 10000
+QuantizationAware: true
+

adaptive-learning/docker-compose.yml

@@ -0,0 +1,32 @@
+version: '2.3'
+services:
+  teacher:
+    build:
+      context: ./teachers/
+      dockerfile: iterdet-teacher.Dockerfile
+    stdin_open: true
+    tty: true
+    container_name: teacher-container
+    volumes:
+      - ../../:/repo
+
+    runtime: nvidia
+    environment:
+      - NVIDIA_VISIBLE_DEVICES=0
+
+  student:
+    build:
+      context: ./students/
+      dockerfile: ssd-mobilenet-v2-student.Dockerfile
+    stdin_open: true
+    tty: true
+    container_name: student-container
+    volumes:
+    - ../../:/repo
+    ports:
+      - "2029:6006"
+    runtime: nvidia
+    environment:
+      - NVIDIA_VISIBLE_DEVICES=0
+      - PYTHONPATH=$PYTHONPATH:/models/research:/models/research/slim
+      - TF_FORCE_GPU_ALLOW_GROWTH=true

adaptive-learning/download_sample_video.sh

@@ -0,0 +1 @@
+wget https://social-distancing-data.s3.amazonaws.com/softbio_vid.mp4 -O data/softbio_vid.mp4

adaptive-learning/students/apply_configs.py

@@ -0,0 +1,46 @@
+import argparse
+import os
+import sys
+from pathlib import Path
+import logging
+
+sys.path.append("../")
+from configs.config_engine import ConfigEngine
+
+import tensorflow as tf
+from google.protobuf import text_format
+from object_detection.protos import pipeline_pb2
+
+
+def main(args):
+    """
+    This script will change some items TensorFlow Object Detection API training config file base on adaptive learning
+    config.ini
+    """
+    pipeline_config = pipeline_pb2.TrainEvalPipelineConfig()
+
+    with tf.gfile.GFile(args.pipeline, "r") as f:
+        proto_str = f.read()
+        text_format.Merge(proto_str, pipeline_config)
+    config = ConfigEngine(args.config)
+    pipeline_config.train_config.batch_size = int(config.get_section_dict('Student')['BatchSize'])
+    data_dir = Path(config.get_section_dict('Teacher')['ImagePath']).parent
+    pipeline_config.train_input_reader.tf_record_input_reader.input_path[:] = [os.path.join(str(data_dir),
+                                                                                            "tfrecords",
+                                                                                            "train.record")]
+    pipeline_config.eval_input_reader[:][0].tf_record_input_reader.input_path[:] = [os.path.join(str(data_dir),
+                                                                                                 "tfrecords",
+                                                                                                 "val.record")]
+
+    config_text = text_format.MessageToString(pipeline_config)
+    with tf.gfile.Open(args.pipeline, "wb") as f:
+        f.write(config_text)
+    logging.info("The config pipeline has been changes")
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description='')
+    parser.add_argument('--pipeline', required=True, type=str)
+    parser.add_argument('--config', required=True, type=str)
+    args = parser.parse_args()
+    main(args)