knowit2020/slides.html

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <meta name="generator" content="pandoc">
  <meta name="author" content="Jon Nordby jon@soundsensing.no">
  <meta name="dcterms.date" content="2020-02-27">
  <title>Artificial Summit, Knowit, Oslo: Classifying sound using Machine Learning</title>
  <meta name="apple-mobile-web-app-capable" content="yes">
  <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">
  <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no, minimal-ui">
  <link rel="stylesheet" href="reveal.js/css/reset.css">
  <link rel="stylesheet" href="reveal.js/css/reveal.css">
  <style>
    code{white-space: pre-wrap;}
    span.smallcaps{font-variant: small-caps;}
    span.underline{text-decoration: underline;}
    div.column{display: inline-block; vertical-align: top; width: 50%;}
    div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
  </style>
  <link rel="stylesheet" href="reveal.js/css/theme/white.css" id="theme">
  <link rel="stylesheet" href="style.css"/>
  <!-- Printing and PDF exports -->
  <script>
    var link = document.createElement( 'link' );
    link.rel = 'stylesheet';
    link.type = 'text/css';
    link.href = window.location.search.match( /print-pdf/gi ) ? 'reveal.js/css/print/pdf.css' : 'reveal.js/css/print/paper.css';
    document.getElementsByTagName( 'head' )[0].appendChild( link );
  </script>
  <!--[if lt IE 9]>
  <script src="reveal.js/lib/js/html5shiv.js"></script>
  <![endif]-->
</head>
<body>
  <div class="reveal">
    <div class="slides">

<section id="title-slide">
  <h1 class="title">Classifying sound using Machine Learning</h1>
  <p class="author">Jon Nordby <a href="mailto:jon@soundsensing.no" class="email">jon@soundsensing.no</a></p>
  <p class="date">February 27, 2020</p>
</section>

<section><section id="introduction" class="title-slide slide level1"><h1>Introduction</h1></section>
<section id="jon-nordby" class="slide level2">
<h2>Jon Nordby</h2>
<p>Internet of Things specialist</p>
<ul>
<li>B.Eng in <strong>Electronics</strong></li>
<li>10 years as <strong>Software</strong> developer. <strong>Embedded</strong> + <strong>Web</strong></li>
<li>M. Sc in <strong>Data</strong> Science</li>
</ul>
<p>Now:</p>
<ul>
<li>CTO at Soundsensing</li>
</ul>
</section>
<section id="thesis" class="slide level2">
<h2>Thesis</h2>
<blockquote>
<p>Environmental Sound Classification on Microcontrollers using Convolutional Neural Networks</p>
</blockquote>
<figure>
<img data-src="./img/thesis.png" style="width:30.0%" alt="" /><figcaption>Report &amp; Code: https://github.com/jonnor/ESC-CNN-microcontroller</figcaption>
</figure>
</section>
<section id="soundsensing" class="slide level2">
<h2>Soundsensing</h2>
<p><img data-src="./img/soundsensing-withlogo.png" style="width:100.0%" /></p>
<aside class="notes">
<p>Provide <strong>Noise Monitoring</strong> and Audio <strong>Condition Monitoring</strong> solutions that are used in Real-Estate, Industry, and Smart Cities.</p>
<p>Perform Machine Learning for sound classification <strong>on sensor</strong>.</p>
</aside>
</section>
<section id="dashboard" class="slide level2">
<h2>Dashboard</h2>
<figure>
<img data-src="img/what-we-do.png" alt="" /><figcaption>Pilot projects with customers Now - 2020</figcaption>
</figure>
</section>
<section id="goal" class="slide level2">
<h2>Goal</h2>
<p>The goals of this talk</p>
<blockquote>
<p>you as Developers, understand:</p>
<p>possibilities and applications of Audio ML</p>
<p>overall workflow of creating an Audio Classification solution</p>
<p>what Soundsensing provides in this area</p>
</blockquote>
</section></section>
<section><section id="background" class="title-slide slide level1"><h1>Background</h1></section>
<section id="why-audio-classification" class="slide level2">
<h2>Why Audio Classification</h2>
<ul>
<li>Rich source of information</li>
<li>Any physical motion creates sound</li>
<li>Sound spreads in a space</li>
<li>Enables <em>non invasive sensing</em></li>
<li>Great compliment to image/video</li>
<li>Humans use our hearing usefully in many tasks</li>
<li>Machines can now reach similar performance</li>
</ul>
</section>
<section id="applications" class="slide level2">
<h2>Applications</h2>
<p>Audio sub-fields</p>
<ul>
<li>Speech Recognition. Keyword spotting.</li>
<li>Music Analysis. Genre classification.</li>
<li><strong>General</strong> / other</li>
</ul>
<p>Examples</p>
<ul>
<li>Eco-acoustics. Analyze bird migrations</li>
<li>Wildlife preservation. Detect poachers in protected areas</li>
<li>Manufacturing Quality Control. Testing electric car seat motors</li>
<li>Security: Highlighting CCTV feeds with verbal agression</li>
<li>Medical. Detect heart murmurs</li>
<li>Process industry. Advance process once audible event happens (popcorn)</li>
</ul>
</section>
<section id="neural-network-co-processors" class="slide level2">
<h2>Neural Network co-processors</h2>
<figure>
<img data-src="img/ST-Orlando-SoC.png" style="width:25.0%" alt="" /><figcaption>Project Orlando (ST Microelectronics), expected 2020</figcaption>
</figure>
<p>Expected 10x power efficiency increases.</p>
<aside class="notes">
<p>2.9 TOPS/W. AlexNet, 1000 classes, 10 FPS. 41 mWatt</p>
<p>Audio models probably <strong>&lt; 1 mWatt</strong>.</p>
<p>https://www.latticesemi.com/Blog/2019/05/17/18/25/sensAI</p>
</aside>
<!--

## Outline

TODO: is it needed?

- 
- Digital sound. A primer
- Audio Classification basics
- Tips & Tricks
- Pointers to more information
-->
<!--
FIXME: explain briefly Digital Sound
-->
</section></section>
<section><section id="making-an-audio-ml-solution" class="title-slide slide level1"><h1>Making an Audio ML solution</h1></section>
<section id="overall-process" class="slide level2">
<h2>Overall process</h2>
<ol type="1">
<li>Problem definition</li>
<li>Data collection</li>
<li>Data labeling</li>
<li>Training setup</li>
<li>Feature representation</li>
<li>Model</li>
<li>Evaluation</li>
<li>Deployment</li>
</ol>
</section>
<section id="example-task" class="slide level2">
<h2>Example task</h2>
<p>Noise Classification in Urban environments. AKA <em>“Environmental Sound Classification”</em></p>
<blockquote>
<p>Given an audio signal of environmental sounds,</p>
<p>determine which class it belongs to</p>
</blockquote>
<ul>
<li>Widely researched. 1000 hits on Google Scholar</li>
<li>Open Datasets. Urbansound8k (10 classes), ESC-50, AudioSet (632 classes)</li>
<li>2017: Human-level performance on ESC-50</li>
</ul>
<aside class="notes">
<p>https://github.com/karoldvl/ESC-50</p>
</aside>
</section>
<section id="supervised-learning" class="slide level2">
<h2>Supervised Learning</h2>
<figure>
<img data-src="./img/supervised-principle.svg" style="width:70.0%" alt="" /><figcaption>In principle…</figcaption>
</figure>
<aside class="notes">
<p>Supervised learning: using (large amounts of) <strong>labeled data</strong>, for training. Optimizes a particular <strong>metric</strong>. Estimated on a parts of the dataset.</p>
<p>In principle, needs no more guidance. In practice, we need to peek inside the black box.</p>
<p>AutoML. Field of creating that magical black-box.</p>
</aside>
</section>
<section id="learning-process" class="slide level2">
<h2>Learning process</h2>
<p><img data-src="./img/learning-process.svg" style="width:80.0%" /></p>
<aside class="notes">
<p>Inputs.</p>
<ul>
<li>Labeled dataset. Many audio samples, each <strong>already labeled</strong> (by humans) with the associated class</li>
<li>Model architecture. An untrained model for recognizing sounds (more generally).</li>
<li>Goal specification. The metric we want to optimize for. Ex: average accuracy</li>
</ul>
<p>Outputs:</p>
<ul>
<li>A <strong>trained model</strong>. That can classify Environmental Sounds with high accuracy. Hopefully also sounds similar to, but not from the Urbansound8k dataset.</li>
<li>Results. Metrics, plots of how well the model did.</li>
</ul>
<p>How to get dataset? See if there are publically available dataset. General rule. 1k samples per class. But, tricks exist for cases where there is less data available. “Low resource” datasets.</p>
<p>What kind of model architecture to use? Find the most similar usecase to yours. Start simple!</p>
<p>How to specify goal? Analyze your business/usecase. What aspect of model performance are critical. What kind of errors are more acceptable?</p>
</aside>
</section>
<section id="audio-classification" class="slide level2">
<h2>Audio Classification</h2>
<blockquote>
<p>Given an audio clip</p>
<p>with some sounds</p>
<p>determine which <em>class</em> it is</p>
</blockquote>
<p>Classification simplifications</p>
<ul>
<li>Single output. One class at a time</li>
<li>Discrete. Exists or not</li>
<li>Closed set. Must be known class</li>
</ul>
<aside class="notes">
<p>Classification. Each input is mapped to a single discrete output. Only a single class sound per audio clip.</p>
<p>Alt: multi-label “tagging”</p>
<p>Closed-set. Sound has to be one of the N defined classes. Cannot handle out-of-domain inputs.</p>
<p>Alt: open-set</p>
<p>Ok, so how do we realize such a system?</p>
</aside>
<!--
Other Audio ML tasks

## Audio Event Detection

> Return: time something occurred.

* Ex: "Bird singing started", "Bird singing stopped"
* Classification-as-detection. Classifier on short time-frames
* Monophonic: Returns most prominent event

Aka: Onset detection

::: notes

- Avoid time-shift augmentation

:::

## Anomaly Detection

TODO: define AD task

-->
</section>
<section id="data-collection" class="slide level2">
<h2>Data collection</h2>
<p>Key challenges</p>
<ul>
<li>Ensuring representativeness</li>
<li>Ensuring coverage</li>
<li>Maintaining structure</li>
<li>Capturing relevant metadata</li>
<li>Maintaining privacy</li>
</ul>
<aside class="notes">
<p><code>FIXME: image of our Data document</code></p>
<p>Personally identifyable information. GDPR Sensitive data Privacy</p>
</aside>
</section>
<section id="data-management" class="slide level2">
<h2>Data management</h2>
<figure>
<img data-src="img/plans.png" style="width:80.0%" alt="" /><figcaption>Data management plans</figcaption>
</figure>
<p>Best practice: Design the process, document in a <strong>protocol</strong></p>
</section>
<section id="data-requirements" class="slide level2">
<h2>Data Requirements</h2>
<p>Depends on problem difficulty</p>
<ul>
<li>Number of classes</li>
<li>Variation inside class</li>
<li>Distance between classes</li>
<li>Other sounds, outside classes</li>
</ul>
<p>Targets</p>
<ul>
<li>Must: &gt;100 labeled instances per class</li>
<li>Want: &gt;1000 labeled instances per class</li>
</ul>
<p>Urbansound8k: 10 classes, 11 hours of <strong>annotated</strong> audio</p>
<aside class="notes">
<p>Roest is <code>&lt;1</code> hour of audio. Binary classification Clear separation between class1 and class0 Quite little variation in the classes Controlled environment</p>
</aside>
</section>
<section id="data-labeling" class="slide level2">
<h2>Data labeling</h2>
<p>Challenge: Keeping quality high, and costs low</p>
<ul>
<li>What is the <em>human level performance</em>?</li>
<li>Annotator <em>self-agreement</em></li>
<li>Inter-annotator <em>agreement</em></li>
</ul>
<p>How to label</p>
<ul>
<li>Outsource. Data labeling services, Mechanical Turk, ..</li>
<li>Crowdsource. From the public. From your users?</li>
<li>Inhouse. Dedicated resource? Part of DS team?</li>
<li>Automated. Other datasources? Existing models?</li>
</ul>
<aside class="notes">
<p>Make sure at least validation and testsets are good Run chesks. Estimate quality</p>
<p>Expertice required? Domain knowledge?</p>
</aside>
</section>
<section id="annotation-tools" class="slide level2">
<h2>Annotation tools</h2>
<figure>
<img data-src="img/soundsensing-audioannotator.png" style="width:80.0%" alt="" /><figcaption>Labeling audio data</figcaption>
</figure>
<aside class="notes">
<p><code>FIXME: image of our tool</code></p>
<p>Audacity Audio Annotator …</p>
</aside>
</section>
<section id="curated-dataset" class="slide level2">
<h2>Curated dataset</h2>
<figure>
<img data-src="img/urbansound8k-examples.png" style="width:80.0%" alt="" /><figcaption>Labeled audio data</figcaption>
</figure>
<aside class="notes">
<p><code>FIXME: image of our tool</code></p>
</aside>
<!--

## Training setup

::: notes

For each annotated section
Get the audio
Compute features on it
Mark it with the label

Pretty standard supervised ML setup
As in image classification etc

:::

-->
</section>
<section id="pipeline" class="slide level2">
<h2>Pipeline</h2>
<p><img data-src="img/classification-pipeline.png" data-max-height="100%" /></p>
<aside class="notes">
<p>TODO:</p>
</aside>
<!--
FIXME: explain briefly Convolutional Neural Network
-->
</section>
<section id="convolutional-neural-network" class="slide level2">
<h2>Convolutional Neural Network</h2>
<figure>
<img data-src="img/cnn.png" style="width:70.0%" alt="" /><figcaption>Best-in-class for image recognition tasks</figcaption>
</figure>
</section>
<section id="model" class="slide level2">
<h2>Model</h2>
<!--
Based on SB-CNN (Salamon+Bello, 2016)
-->
<p><img data-src="img/models.svg" style="width:70.0%" /></p>
<aside class="notes">
<p>Baseline from SB-CNN</p>
<p>Few modifications</p>
<ul>
<li>Uses smaller input feature representation</li>
<li>Reduced downsample factor to accommodate</li>
</ul>
<p>CONV = entry point for trying different convolution operators</p>
</aside>
<!--
## Evaluation

Performance metrics

-->
</section>
<section id="evaluation" class="slide level2">
<h2>Evaluation</h2>
<p><img data-src="img/results.png" style="width:100.0%" /></p>
</section></section>
<section><section id="summary" class="title-slide slide level1"><h1>Summary</h1></section>
<section id="standard-models-exist" class="slide level2">
<h2>Standard models exist</h2>
<p>Try the standard audio pipeline, it often does OK.</p>
<ul>
<li>Fixed-length analysis windows</li>
<li>Use log-mel spectrograms as features</li>
<li>Convolutional Neural Network as classifier</li>
<li>Aggregate prediction from each window</li>
</ul>
<p>All available as open source solutions.</p>
</section>
<section id="data-collection-is-not-magic" class="slide level2">
<h2>Data collection is not magic</h2>
<ul>
<li>Structure collection upfront</li>
<li>Budget resources for collection and labeling</li>
<li>Integrate quality checking</li>
<li>Build it up gradually</li>
</ul>
<p>Doable, but takes time!</p>
</section>
<section id="now-what" class="slide level2">
<h2>Now what</h2>
<p>So have a model that performs Audio Classification <strong>on our PC</strong>.</p>
<p>But we want to monitor a real-world phenomenon.</p>
<p>How to deploy this?</p>
</section></section>
<section><section id="deploying" class="title-slide slide level1"><h1>Deploying</h1></section>
<section id="architectures" class="slide level2">
<h2>Architectures</h2>
<p><img data-src="img/sensornetworks.png" style="width:70.0%" /></p>
<!--
FIXME: 
-->
</section>
<section id="soundsensing-platform" class="slide level2">
<h2>Soundsensing Platform</h2>
<p><img data-src="img/soundsensing-platform.svg" /></p>
</section>
<section id="demo-video" class="slide level2">
<h2>Demo video</h2>
<iframe src="https://www.youtube.com/embed/KQHQxMG1CZo" controls width="1000" height="700"></iframe>
</iframe>
<aside class="notes">
<p>Recognizing Urban sounds</p>
<p>Environmental Sound Classification</p>
</aside>
</section>
<section id="noise-monitoring-solution" class="slide level2">
<h2>Noise Monitoring solution</h2>
<figure>
<img data-src="img/what-we-do.png" alt="" /><figcaption>Pilot projects with customers Now - 2020</figcaption>
</figure>
</section></section>
<section><section id="outro" class="title-slide slide level1"><h1>Outro</h1></section>
<section id="partner-opportunities" class="slide level2">
<h2>Partner opportunities</h2>
<p>We are building a partner network</p>
<ul>
<li>Partner role: Deliver solutions and integration on platform</li>
<li>Especially cases that require a lot of custom work</li>
</ul>
<p>Email: <a href="mailto:jon@soundsensing.no" class="email">jon@soundsensing.no</a></p>
</section>
<section id="employment-opportunities" class="slide level2">
<h2>Employment opportunities</h2>
<p>Think that this sounds cool to work on?</p>
<ul>
<li>Internships in Data Science now</li>
<li>Hiring 2 developers in 2020</li>
</ul>
<p>Email: <a href="mailto:jon@soundsensing.no" class="email">jon@soundsensing.no</a></p>
</section>
<section id="investment-opportunities" class="slide level2">
<h2>Investment opportunities</h2>
<p>Want to invest in a Machine Learning and Internet of Things startup?</p>
<ul>
<li>Funding round open now</li>
<li>60% comitted</li>
<li>Some open slots still</li>
</ul>
<p>Email: <a href="mailto:ole@soundsensing.no" class="email">ole@soundsensing.no</a></p>
</section>
<section id="more-resources" class="slide level2">
<h2>More resources</h2>
<p>Machine Hearing. ML on Audio</p>
<ul>
<li><a href="https://github.com/jonnor/machinehearing">github.com/jonnor/machinehearing</a></li>
</ul>
<p>Machine Learning for Embedded / IoT</p>
<ul>
<li><a href="https://github.com/jonnor/embeddedml">github.com/jonnor/embeddedml</a></li>
</ul>
<p>Thesis Report &amp; Code</p>
<ul>
<li><a href="https://github.com/jonnor/ESC-CNN-microcontroller">github.com/jonnor/ESC-CNN-microcontroller</a></li>
</ul>
<p>Soundsensing</p>
<ul>
<li><a href="https://soundsensing.no">soundsensing.no</a></li>
</ul>
</section>
<section id="questions" class="slide level2">
<h2>Questions</h2>
<h1 style="padding: 100px">
?
</h1>
<p>Email: <a href="mailto:jon@soundsensing.no" class="email">jon@soundsensing.no</a></p>
</section></section>
<section id="bonus" class="title-slide slide level1"><h1>BONUS</h1></section>
<section><section id="thesis-results" class="title-slide slide level1"><h1>Thesis results</h1></section>
<section id="model-comparison" class="slide level2">
<h2>Model comparison</h2>
<p><img data-src="img/models_accuracy.png" style="width:100.0%" /></p>
<aside class="notes">
<ul>
<li>Baseline relative to SB-CNN and LD-CNN is down from 79% to 73% Expected because poorer input representation. Much lower overlap</li>
</ul>
</aside>
</section>
<section id="list-of-results" class="slide level2">
<h2>List of results</h2>
<p><img data-src="img/results.png" style="width:100.0%" /></p>
</section>
<section id="confusion" class="slide level2">
<h2>Confusion</h2>
<p><img data-src="img/confusion_test.png" style="width:70.0%" /></p>
</section>
<section id="grouped-classification" class="slide level2">
<h2>Grouped classification</h2>
<p><img data-src="img/grouped_confusion_test_foreground.png" style="width:60.0%" /></p>
<p>Foreground-only</p>
</section>
<section id="unknown-class" class="slide level2">
<h2>Unknown class</h2>
<p><img data-src="img/unknown-class.png" style="width:100.0%" /></p>
<aside class="notes">
<p>Idea: If confidence of model is low, consider it as “unknown”</p>
<ul>
<li>Left: Histogram of correct/incorrect predictions</li>
<li>Right: Precision/recall curves</li>
<li>Precision improves at expense of recall</li>
<li>90%+ precision possible at 40% recall</li>
</ul>
<p>Usefulness:</p>
<ul>
<li>Avoids making decisions on poor grounds</li>
<li>“Unknown” samples good candidates for labeling-&gt;dataset. Active Learning</li>
<li>Low recall not a problem? Data is abundant, 15 samples a 4 seconds per minute per sensor</li>
</ul>
</aside>
</section></section>
<section><section id="background-1" class="title-slide slide level1"><h1>Background</h1></section>
<section id="mel-spectrogram" class="slide level2">
<h2>Mel-spectrogram</h2>
<p><img data-src="img/spectrograms.svg" /></p>
</section>
<section id="noise-pollution" class="slide level2">
<h2>Noise Pollution</h2>
<p>Reduces health due to stress and loss of sleep</p>
<p>In Norway</p>
<ul>
<li>1.9 million affected by road noise (2014, SSB)</li>
<li>10’000 healty years lost per year (Folkehelseinstituttet)</li>
</ul>
<p>In Europe</p>
<ul>
<li>13 million suffering from sleep disturbance (EEA)</li>
<li>900’000 DALY lost (WHO)</li>
</ul>
<aside class="notes">
<p>1.9 million https://www.ssb.no/natur-og-miljo/artikler-og-publikasjoner/flere-nordmenn-utsatt-for-stoy</p>
<p>1999: 1.2 million</p>
<p>10 245 tapte friske leveår i Norge hvert år https://www.miljostatus.no/tema/stoy/stoy-og-helse/</p>
<p>https://www.eea.europa.eu/themes/human/noise/noise-2</p>
<p>Burden of Disease WHO http://www.euro.who.int/__data/assets/pdf_file/0008/136466/e94888.pdf</p>
</aside>
</section>
<section id="noise-mapping" class="slide level2">
<h2>Noise Mapping</h2>
<p>Simulation only, no direct measurements</p>
<p><img data-src="img/stoykart.png" /></p>
<aside class="notes">
<ul>
<li>Known sources</li>
<li>Yearly average value</li>
<li>Updated every 5 years</li>
<li>Low data quality. Ex: communal roads</li>
</ul>
<p>Image: https://www.regjeringen.no/no/tema/plan-bygg-og-eiendom/plan–og-bygningsloven/plan/kunnskapsgrunnlaget-i-planlegging/statistikk-i-plan/id2396747/</p>
</aside>
</section></section>
<section><section id="audio-machine-learning-on-low-power-sensors" class="title-slide slide level1"><h1>Audio Machine Learning on low-power sensors</h1></section>
<section id="wireless-sensor-networks" class="slide level2">
<h2>Wireless Sensor Networks</h2>
<ul>
<li>Want: Wide and dense coverage</li>
<li>Need: Sensors need to be low-cost</li>
<li><strong>Opportunity</strong>: Wireless reduces costs</li>
<li><strong>Challenge</strong>: Power consumption</li>
</ul>
<aside class="notes">
<ul>
<li>No network cabling, no power cabling</li>
<li>No site infrastructure needed</li>
<li>Less invasive</li>
<li>Fewer approvals needed</li>
<li>Temporary installs feasible</li>
<li>Mobile sensors possible</li>
</ul>
<p>Electrician is 750 NOK/hour</p>
<p>Image: https://www.nti-audio.com/en/applications/noise-measurement/unattended-monitoring</p>
</aside>
</section>
<section id="what-do-you-mean-by-low-power" class="slide level2">
<h2>What do you mean by low-power?</h2>
<p>Want: 1 year lifetime for palm-sized battery</p>
<p>Need: <code>&lt;1mW</code> system power</p>
</section>
<section id="general-purpose-microcontroller" class="slide level2">
<h2>General purpose microcontroller</h2>
<p><img data-src="img/cortexM4.png" style="width:40.0%" /></p>
<p>STM32L4 @ 80 MHz. Approx <strong>10 mW</strong>.</p>
<ul>
<li>TensorFlow Lite for Microcontrollers (Google)</li>
<li>ST X-CUBE-AI (ST Microelectronics)</li>
</ul>
</section>
<section id="fpga" class="slide level2">
<h2>FPGA</h2>
<figure>
<img data-src="img/iCE40UltraPlus.png" style="width:50.0%" alt="" /><figcaption>Lattice ICE40 UltraPlus with Lattice sensAI</figcaption>
</figure>
<p>Human presence detection. VGG8 on 64x64 RGB image, 5 FPS: 7 mW.</p>
<p>Audio ML approx <strong>1 mW</strong></p>
</section>
<section id="neural-network-co-processors-micro" class="slide level2">
<h2>Neural Network co-processors micro</h2>
<figure>
<img data-src="img/ST-Orlando-SoC.png" style="width:25.0%" alt="" /><figcaption>Project Orlando (ST Microelectronics), expected 2020</figcaption>
</figure>
<p>2.9 TOPS/W. AlexNet, 1000 classes, 10 FPS. 41 mWatt</p>
<p>Audio models probably <strong>&lt; 1 mWatt</strong>.</p>
<aside class="notes">
<p>https://www.latticesemi.com/Blog/2019/05/17/18/25/sensAI</p>
</aside>
</section>
<section id="urbansound8k" class="slide level2">
<h2>Urbansound8k</h2>
<p><img data-src="img/urbansound8k-examples.png" style="width:100.0%" /></p>
<aside class="notes">
<p>Classes from an urban sound taxonomy, based on noise complains in New York city</p>
<p>Most sounds around 4 seconds. Some classes around 1 second</p>
<p>Foreground/background</p>
</aside>
</section>
<section id="environmental-sound-classification-on-edge" class="slide level2">
<h2>Environmental Sound Classification on edge</h2>
</section>
<section id="existing-work" class="slide level2">
<h2>Existing work</h2>
<ul>
<li>Convolutional Neural Networks dominate</li>
<li>Techniques come from image classification</li>
<li>Mel-spectrogram input standard</li>
<li>End2end models: getting close in accuracy</li>
<li>“Edge ML” focused on mobile-phone class HW</li>
<li>“Tiny ML” (sensors) just starting</li>
</ul>
<aside class="notes">
<ul>
<li>Efficient Keyword-Spotting</li>
<li>Efficient (image) CNNs</li>
<li>Efficient ESC-CNN</li>
</ul>
<p>ESC-CNN</p>
<ul>
<li>23 papers reviewed in detail</li>
<li>10 referenced in thesis</li>
<li>Only 4 consider computational efficiency</li>
</ul>
</aside>
</section>
<section id="model-requirements" class="slide level2">
<h2>Model requirements</h2>
<p>With 50% of STM32L476 capacity:</p>
<ul>
<li>64 kB RAM</li>
<li>512 kB FLASH memory</li>
<li>4.5 M MACC/second</li>
</ul>
<aside class="notes">
<ul>
<li>RAM: 1000x 64 MB</li>
<li>PROGMEM: 1000x 512 MB</li>
<li>CPU: 1000x 5 GFLOPS</li>
<li>GPU: 1000’000X 5 TFLOPS</li>
</ul>
</aside>
</section>
<section id="existing-models" class="slide level2">
<h2>Existing models</h2>
<figure>
<img data-src="img/urbansound8k-existing-models-logmel.png" style="width:100.0%" alt="" /><figcaption>Green: Feasible region</figcaption>
</figure>
<p>eGRU: running on ARM Cortex-M0 microcontroller, accuracy 61% with <strong>non-standard</strong> evaluation</p>
<aside class="notes">
<p>Assuming no overlap. Most models use very high overlap, 100X higher compute</p>
</aside>
</section></section>
<section><section id="research-topics" class="title-slide slide level1"><h1>Research Topics</h1></section>
<section id="waveform-input-to-model" class="slide level2">
<h2>Waveform input to model</h2>
<ul>
<li>Preprocessing. Mel-spectrogram: <strong>60</strong> milliseconds</li>
<li>CNN. Stride-DS-24: <strong>81</strong> milliseconds</li>
<li>With quantization, spectrogram conversion is the bottleneck!</li>
<li>Convolutions can be used to learn a Time-Frequency transformation.</li>
</ul>
<p>Can this be faster than the standard FFT? And still perform well?</p>
<aside class="notes">
<ul>
<li>Especially interesting with CNN hardware acceleration.</li>
</ul>
</aside>
</section>
<section id="on-sensor-inference-challenges" class="slide level2">
<h2>On-sensor inference challenges</h2>
<ul>
<li>Reducing power consumption. Adaptive sampling</li>
<li>Efficient training data collection in WSN. Active Learning?</li>
<li>Real-life performance evaluations. Out-of-domain samples</li>
</ul>
<aside class="notes">
<p>TODO: Add few more projects here. From research document</p>
</aside>
</section></section>
<section><section id="shrinking-a-convolutional-neural-network" class="title-slide slide level1"><h1>Shrinking a Convolutional Neural Network</h1></section>
<section id="reduce-input-dimensionality" class="slide level2">
<h2>Reduce input dimensionality</h2>
<p><img data-src="img/input-size.svg" style="width:70.0%" /></p>
<ul>
<li>Lower frequency range</li>
<li>Lower frequency resolution</li>
<li>Lower time duration in window</li>
<li>Lower time resolution</li>
</ul>
<aside class="notes">
<p>Directly limits time and RAM use first few layers.</p>
<p>Follow-on effects. A simpler input representation is (hopefully) easier to learn allowing for a simpler model</p>
<p>TODO: make a picture illustrating this</p>
</aside>
</section>
<section id="reduce-overlap" class="slide level2">
<h2>Reduce overlap</h2>
<p><img data-src="img/framing.png" style="width:80.0%" /></p>
<p>Models in literature use 95% overlap or more. 20x penalty in inference time!</p>
<p>Often low performance benefit. Use 0% (1x) or 50% (2x).</p>
<!--
## Regular 2D-convolution

![](img/convolution-2d.png){width=100%}

::: notes

TODO: illustrate the cubical nature. Many channel

:::
-->
</section>
<section id="depthwise-separable-convolution" class="slide level2">
<h2>Depthwise-separable Convolution</h2>
<p><img data-src="img/depthwise-separable-convolution.png" style="width:90.0%" /></p>
<p>MobileNet, “Hello Edge”, AclNet. 3x3 kernel,64 filters: 7.5x speedup</p>
<aside class="notes">
<ul>
<li>Much fewer operations</li>
<li>Less expressive - but regularization effect can be beneficial</li>
</ul>
</aside>
</section>
<section id="spatially-separable-convolution" class="slide level2">
<h2>Spatially-separable Convolution</h2>
<p><img data-src="img/spatially-separable-convolution.png" style="width:90.0%" /></p>
<p>EffNet, LD-CNN. 5x5 kernel: 2.5x speedup</p>
</section>
<section id="downsampling-using-max-pooling" class="slide level2">
<h2>Downsampling using max-pooling</h2>
<p><img data-src="img/maxpooling.png" style="width:100.0%" /></p>
<p>Wasteful? Computing convolutions, then throwing away 3/4 of results!</p>
</section>
<section id="downsampling-using-strided-convolution" class="slide level2">
<h2>Downsampling using strided convolution</h2>
<p><img data-src="img/strided-convolution.png" style="width:100.0%" /></p>
<p>Striding means fewer computations and “learned” downsampling</p>
</section>
<section id="model-comparison-1" class="slide level2">
<h2>Model comparison</h2>
<p><img data-src="img/models_accuracy.png" style="width:100.0%" /></p>
<aside class="notes">
<ul>
<li>Baseline relative to SB-CNN and LD-CNN is down from 79% to 73% Expected because poorer input representation. Much lower overlap</li>
</ul>
</aside>
</section>
<section id="performance-vs-compute" class="slide level2">
<h2>Performance vs compute</h2>
<p><img data-src="img/models_efficiency.png" style="width:100.0%" /></p>
<aside class="notes">
<ul>
<li>Performance of Strided-DS-24 similar to Baseline despite 12x the CPU use</li>
<li>Suprising? Stride alone worse than Strided-DS-24</li>
<li>Bottleneck and EffNet performed poorly</li>
<li>Practical speedup not linear with MACC</li>
</ul>
</aside>
<!--

-->
</section>
<section id="quantization" class="slide level2">
<h2>Quantization</h2>
<p>Inference can often use 8 bit integers instead of 32 bit floats</p>
<ul>
<li>1/4 the size for weights (FLASH) and activations (RAM)</li>
<li>8bit <strong>SIMD</strong> on ARM Cortex M4F: 1/4 the inference time</li>
<li>Supported in X-CUBE-AI 4.x (July 2019)</li>
</ul>
<!--

## Stronger training process 

- Data Augmentation. *Mixup*, *SpecAugment*
- Transfer Learning on more data. *AudioSet*

-->
<aside class="notes">
<p>EnvNet-v2 got 78.3% on Urbansound8k with 16 kHz</p>
</aside>
</section></section>
    </div>
  </div>

  <script src="reveal.js/js/reveal.js"></script>

  <script>

      // Full list of configuration options available at:
      // https://github.com/hakimel/reveal.js#configuration
      Reveal.initialize({
        // Push each slide change to the browser history
        history: true,
        // The "normal" size of the presentation, aspect ratio will be preserved
        // when the presentation is scaled to fit different resolutions. Can be
        // specified using percentage units.
        width: 1920,
        height: 1080,
        // Factor of the display size that should remain empty around the content
        margin: 0,

        // Optional reveal.js plugins
        dependencies: [
          { src: 'reveal.js/lib/js/classList.js', condition: function() { return !document.body.classList; } },
          { src: 'reveal.js/plugin/zoom-js/zoom.js', async: true },
          { src: 'reveal.js/plugin/notes/notes.js', async: true }
        ]
      });
    </script>
    </body>
</html>