Skip to content

Home

Jump to bottom
Andrew Liu edited this page Apr 26, 2017 · 33 revisions

Welcome to the azure-documentdb-spark wiki!

This wiki contains the following resources for your reference:

Introduction

This project provides a client library that allows Azure DocumentDB to act as an input source or output sink for Spark jobs. Fast connectivity between Apache Spark and Azure DocumentDB accelerates your ability to solve your fast moving Data Sciences problems where your data can be quickly persisted and retrieved using Azure DocumentDB. With the Spark to DocumentDB connector, you can more easily solve scenarios including (but not limited to) blazing fast IoT scenarios, update-able columns when performing analytics, push-down predicate filtering, and performing advanced analytics to data sciences against your fast changing data against a geo-replicated managed document store with guaranteed SLAs for consistency, availability, low latency, and throughput.

This package is provided as a technical preview.

Common Scenarios

Common scenarios to use Apache Spark and DocumentDB together include:

  • Distributed Aggregations and Analytics
  • Push-down Predicate Filtering
  • Blazing Fast IoT Scenarios
  • Updateable Columns

Below are more details surrounding the scenario; if you're ready to use azure-documentdb-spark, please refer to the Azure DocumentDB Spark Connector User Guide.

Distributed Aggregations and Analytics

While Azure DocumentDB has aggregations (SUM, MIN, MAX, COUNT, SUM and working on GROUP BY, DISTINCT, etc.), connecting Apache Spark to DocumentDB allows you to easily and quickly perform distributed aggregations which is important for larger implementations. For example, below is a screenshot of calculating a distributed MEDIAN calculation using Apache Spark's PERCENTILE_APPROX function.

select destination, percentile_approx(delay, 0.5) as median_delay 
from df 
where delay < 0 
group by destination 
order by percentile_approx(delay, 0.5)

.

Push-down Predicate Filtering

As noted in the following animated gif, the queries from Apache Spark will push down predicts to Azure DocumentDB and take advantage that DocumentDB indexes every attribute by default.

For example, if you only want to ask for the flights departing from Seattle (SEA), azure-documentdb-spark will:

  • Send the query to Azure DocumentDB
  • As all attributes within Azure DocumentDB are automatically indexed, only the flights pertaining to Seattle will be returned to the Spark worker nodes quickly.
  • This way as you perform your Data Sciences work, you will only transfer the data you need.

.

Blazing Fast IoT Scenarios

Azure DocumentDB is designed for high-throughput, low-latency IoT environments. The animated GIF below refers to the a flights scenario.

Together, you can:

  • Handle high throughput of concurrent alerts (e.g. weather, flight information, global safety alerts, etc.)
  • Send this information downstream for device notifications, RESTful services, etc. (e.g. alert on your phone of an impending flight delay) including the use of change feed
  • At the same time, as you are building up ML models against your data, you can also make sense of the latest information

Updateable Columns

Related to the above noted blazing fast IoT scenarios, let's dive into updateable columns:

As the new piece of information comes in (e.g. the flight delay has changed from 5 min to 30 min), you want to be able to quickly re-run your ML models to reflect this newest information. For example, you can predict the impact of the 30min for all of the downstream flights. This event can be quickly initiated via the Azure DocumentDB Change Feed to refresh your ML models.

Clone this wiki locally