tracking_usage_pres_crug.html

<!DOCTYPE html>
<html lang="" xml:lang="">
  <head>
    <title>tracking_usage_pres_crug.knit</title>
    <meta charset="utf-8" />
    <script src="libs/header-attrs-2.14/header-attrs.js"></script>
    <link href="libs/remark-css-0.0.1/default.css" rel="stylesheet" />
    <link href="libs/remark-css-0.0.1/default-fonts.css" rel="stylesheet" />
    <link href="libs/tile-view-0.2.6/tile-view.css" rel="stylesheet" />
    <script src="libs/tile-view-0.2.6/tile-view.js"></script>
    <link href="libs/animate.css-3.7.2/animate.xaringan.css" rel="stylesheet" />
    <link href="libs/tachyons-4.12.0/tachyons.min.css" rel="stylesheet" />
    <script src="libs/xaringanExtra_fit-screen-0.2.6/fit-screen.js"></script>
    <script type="application/json" id="xaringanExtra-editable-docid">{"id":"x874083a8f5d4428841435e824d90b56","expires":14}</script>
    <script src="libs/himalaya-1.1.0/himalaya.js"></script>
    <script src="libs/js-cookie-3.0.0/js.cookie.js"></script>
    <link href="libs/editable-0.2.6/editable.css" rel="stylesheet" />
    <script src="libs/editable-0.2.6/editable.js"></script>
    <script src="libs/xaringanExtra-webcam-0.0.1/webcam.js"></script>
    <script id="xaringanExtra-webcam-options" type="application/json">{"width":"200","height":"200","margin":"1em"}</script>
    <script src="libs/clipboard-2.0.6/clipboard.min.js"></script>
    <link href="libs/xaringanExtra-clipboard-0.2.6/xaringanExtra-clipboard.css" rel="stylesheet" />
    <script src="libs/xaringanExtra-clipboard-0.2.6/xaringanExtra-clipboard.js"></script>
    <script>window.xaringanExtraClipboard(null, {"button":"Copy Code","success":"Copied!","error":"Press Ctrl+C to Copy"})</script>
    <script src="libs/mark.js-8.11.1/mark.min.js"></script>
    <link href="libs/xaringanExtra-search-0.0.1/search.css" rel="stylesheet" />
    <script src="libs/xaringanExtra-search-0.0.1/search.js"></script>
    <script>window.addEventListener('load', function() { window.xeSearch = new RemarkSearch({"position":"bottom-left","caseSensitive":false,"showIcon":false,"autoSearch":true}) })</script>
    <script src="libs/fabric-4.3.1/fabric.min.js"></script>
    <link href="libs/xaringanExtra-scribble-0.0.1/scribble.css" rel="stylesheet" />
    <script src="libs/xaringanExtra-scribble-0.0.1/scribble.js"></script>
    <script>document.addEventListener('DOMContentLoaded', function() { window.xeScribble = new Scribble({"pen_color":["#FF0000"],"pen_size":3,"eraser_size":30,"palette":[]}) })</script>
    <link href="libs/xaringanExtra-extra-styles-0.2.6/xaringanExtra-extra-styles.css" rel="stylesheet" />
  </head>
  <body>
    <textarea id="source">


layout: false
class: title-slide, inverse, middle, right, main_slide
name: title-slide


# Tracking shiny app usage
## A Reivew of tools
### [CRUG](https://www.cytel.com/)

&lt;br&gt;

Aaron Clark &lt;br&gt; Principal Data Scientist &lt;br&gt; Biogen &lt;br&gt;
2022-08-31 &lt;br&gt;
[bit.ly/crug_aug22](https://bit.ly/crug_aug22) - slides


---
# Learning Shiny 

#### Resources
- [Written and video tutorials](https://shiny.rstudio.com/tutorial/) for users who are new to Shiny
- [Shiny app gallery](https://shiny.rstudio.com/gallery/) to be inspired by and to learn from
- [Articles](https://shiny.rstudio.com/articles/) related to all aspects of Shiny app development
- [Getting help](https://shiny.rstudio.com/articles/help.html) about anything related to R and Shiny

#### Books
- [Mastering Shiny](https://mastering-shiny.org/) by Hadley Wickham
- [Engineering Production-Grade Shiny Apps](https://engineering-shiny.org/) by Colin Fay and others
- [Outstanding User Interfaces with Shiny](https://unleash-shiny.rinterface.com/index.html) by David Granjon

#### Tutorials
- [Shiny tutorial and cheat sheet](http://zevross.com/blog/2016/04/19/r-powered-web-applications-with-shiny-a-tutorial-and-cheat-sheet-with-40-example-apps/) by Zev Ross
- [A gradual introduction to Shiny](https://laderast.github.io/gradual_shiny/) by Ted Laderas et al.
- [Building Shiny apps](https://deanattali.com/blog/building-shiny-apps-tutorial/) – an interactive tutorial by Dean Attali

---

layout: false
class: middle, center
name: what

# What is *user adoption*?

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[
User adoption, sometimes just called onboarding, is the process by which new users become acclimated to a product or service and decide to keep using it. Users only adopt a product if it helps them achieve a goal of theirs.

.tr[
— Jenny Booth
]]

&lt;br&gt;

Source: [mixpanel.com](https://mixpanel.com/blog/how-to-develop-and-measure-a-user-adoption-strategy/#:~:text=User%20adoption%2C%20sometimes%20called%20onboarding,food%20or%20launching%20a%20business)

---

# Why do I care?

* As a shiny developer, we often times spend vast amounts of time programming a user experience everyone can appreciate, so having people use and appreciate your hard work feels good, and it means you've created something truly valuable.


![work too hard](images/work_too_hard.gif)

[gif source](https://media4.giphy.com/media/AYECTMLNS4o67dCoeY/giphy-downsized-large.gif)

---

## If people aren't using your app

You probably...


* misunderstood your stakeholders goals/ needs
* made it too difficult to achieve their goals/ needs
* Any others?


![UI too hard](images/ui_too_difficult.gif)

--

Maybe it means going back to the drawing board to re-assess your approach.

&lt;br&gt;

[gif source](https://media4.giphy.com/media/J2zwN64xc4wgw/200.gif)

---

# The strategy 

&gt; "I want you to spend your precious cognitive resources on the specific details of your data, not on struggling to get R to do what you want. My long term goal is to create a [pit of success](https://quorasessionwithhadleywickham.quora.com/What-is-the-vision-behind-the-tidyverse) where the default path leads to a great result." - Hadley Wickham

&lt;br&gt;

![Pit of Success](images/pit.gif)

--

&lt;br&gt;

How do we do this? 

[gif source](https://i.makeagif.com/media/3-12-2016/m36Cqf.gif)

---

# Measure, measure, measure

* Before writing code &amp; at each iteration of the project, understand users: 
  * motivations
  * needs
  * environments
  * beliefs
  * complaints
  
  
![Measure](images/measure.jpg)
--

* After the project launch, continue assessing.

---
class: inverse, middle, center

# What if...

You have limited communication with end users?

&lt;br&gt;

--

### OR

You aren't getting the whole story via user feedback?

&lt;br&gt;

--

## Trust me, you aren't!
  
---

## Enter: Shiny App Usage Tracking

Every app should have it. Be able to answer questions like:

* Is my app even being used?

--

* How many users are there?

--

* Is a specific person using my app (like the CEO)?

--

* How often are they using it?

--

* How long are they using it?

--

* Which features or workflows are being used most/least?

--

  * Does that align with my expectations?
  
--

  * Do I need to enhance the user friendliness of certain widgets to increase the adoption of features?
  
--

* Do I need to spend extra time marketing my application to my target users, or perhaps market certain features?


---
class: inverse, center, middle

# Tools

There are many, and which one you choose will depend on your end goal


---
# Today's learning objectives:

.pull-left[
- Gain a general working knowledge of 

  - bit.ly (yep)
  
  - Google Analytics

  - Matomo Analytics

  - `{shinylogs}`

  - `{shiny.stats}`
  
  - RSConnect instrumentation data
]

.pull-right[

&lt;br&gt;

.center[&lt;img src="images/tools.png" alt="tools" width="250"/&gt;]
]

--

- Understand when &amp; where to use each tool

--

- Encourage you: use them!


---
class: inverse, center, middle
# bit.ly

Don't overlook the power of monitoring clicks!

---

# bit.ly

What is it?

* Basically a free URL shortener that also counts URL clicks

--

Why use it?


* Know you if users are opening your app

--

* Also great for conferences when demo-ing an app to measure audience engagement

--

* Handy elsewhere - for example: on your app's documentation link(s)!

--

When to use?

* I'd recommend for public-facing URLs, since you don't want to "upload" a private URL to a public site like bit.ly.

---

# How do I create a bit.ly link?

Copy &amp; paste a (long) link into their link shortener, and customize the back half to start collecting stats!

![bit.ly example](images/bitly_example.PNG)

### Easy peasy

---
class: inverse, center, middle
# Google Analytics

Google Analytics is a free service that collects information about who visits your website and what they do while they're there.

--

Best used when you have a public facing app (i.e. shinyapps.io)

---
# Google Analytics: Getting Started

* Create an account (analytics.google.com)

--

* Find your tracking code for your website (aka property)


```js
/*&lt;!-- Global site tag (gtag.js) - Google Analytics --&gt;*/
&lt;script async src="https://www.googletagmanager.com/gtag/js?id=G-ThisIsNotReal"&gt;
&lt;script&gt;
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-ThisIsNotReal');
&lt;/script&gt;
```

---
# Google Analytics: Getting Started (cont'd)

* Add it to the app


```r
# somewhere inside the UI
shiny::tags$head(
  shiny::tags$script(
    src = "https://www.googletagmanager.com/gtag/js?id=G-ThisIsNotReal",
    async = ""
  ),
  shiny::tags$script(
    src = "www/gtag.js"
  )
)
```
---

# Google Analytics: Getting Started (cont'd)

* Redeploy &amp; test

After deployment &amp; visiting shiny app site, you may have to wait about 1 - 2 minutes (max) before data stream flowing into GA report.

&lt;br&gt;

![GA_visits](images/ga_first_time_usage.PNG)
---
class: inverse, center, middle
# Track events with GA

Note: requires some javascript knowledge!

---
# Add an event listener


```js
$('.item').on('click', (event) =&gt; {
  const selectedSpecie = $(event.currentTarget).data().value;
  gtag('event', 'specie_selected', {
    selected_specie: selectedSpecie
  });
});
```

**^^^ Sends the data to GA:**

![Event Tracked](images/ga_event_tracked.PNG)

---
class: inverse, center, middle
# Matomo Analytics

Google Analytics, but open source, offering an on-prem opportunity. I suggest using when you have a private, enterprise app (i.e. internal Shiny Server or RS Connect).

--

Basically the same approach as GA, so we'll skip the "how to".


---
class: inverse, center, middle
# {shinylogs}


---

# `{shinylogs}`

Logging tool for Shiny applications: record inputs or outputs changes, and info about user’s session. All recording is done client-side to not slow down the application and occupy the server.

### Install

```r
# From CRAN
install.packages("shinylogs") 

# From Github
remotes::install_github("dreamRs/shinylogs")
```


---

# `{shinylogs}` Usage

Call the main function `track_usage()` in server part of application, and specify where to write logs:


```r
# UI
use_tracking()

# server
track_usage(storage_mode = store_json(path = "logs/"))
```

--


The function will record :

* **inputs**: each time an input changes, the name, time stamp and value will be saved
* **errors**: errors propagated through outputs
* **outputs**: each time an output is re-generated
* **session**: information about user’s browser and the application


---

# `{shinylogs}` Storage modes 
Six modes are available:

* **store_json()**: store logs has separate JSON files (one by session).

--

* **store_rds()**: store logs has individually RDS files (one by session).

--

* **store_sqlite()**: store logs in a SQLite database.

--

* **store_null()**: don’t write logs on disk, print them in R console.

--

* **store_custom()**: use a custom function to save logs wherever you want.

--

* **store_googledrive()**: store logs(as JSON files) in Googledrive.


---
class: inverse, center, middle
# Deployment with `{shinylogs}`

Each storage mode comes in handy for different deployment environment.

---
Deployment with `{shinylogs}`
# Shiny server, RStudio Connect

On a server, if you want to save logs on disk, don’t forget to set write permission on the folder you want to save logs.

On RStudio Connect, you need to use an absolute path to specify the directory where to save logs. You can find more information here: [Persistent Storage on RStudio Connect](https://support.rstudio.com/hc/en-us/articles/360007981134-Persistent-Storage-on-RStudio-Connect).

---
Deployment with `{shinylogs}`
# shinyapps.io

There’s no persistent data storage! So you can’t save logs as JSON or RDS files, you have to use a **remote storage method**. For example, you can send logs to Google Drive with store_googledrive or use store_custom to send logs wherever you want (dropbox, a remote database, etc…).

To use Google Drive, you’ll need to work with Google’s API and set auth to your account, see [`{gargle}` documentation](https://gargle.r-lib.org/index.html) for examples and how to.

---
Deployment with `{shinylogs}`
# ShinyProxy

With [ShinyProxy](https://www.shinyproxy.io/), you can use a Docker volume to write logs outside of the application container. In `application.yml`, you use can something like this in the specs describing the application:

&gt; container-volumes: [ "/var/log/shinylogs:/root/logs" ]

`/var/log/shinylogs` is a directory on the server where you deploy your applications with ShinyProxy. `/root/logs` is a directory inside your Docker image, it’s the path you can use in `track_usage()`, e.g. :

```
track_usage(
  storage_mode = store_json(path = "/root/logs")
)
```

---
class: inverse, center, middle
# Information recorded by `{shinylogs}`

---
## Session Info

Metadata about the application and the user’s browser:

* **app**: name of the application

* **user**: name of the user (if using Shiny-server pro for example)

* **server_connected**: when application has been launched (server time)

* **sessionid**: a session ID to match the session with other recorded tables

* **server_disconnected**: when the application was disconnected (server time)

* **user_agent**: browser user-agent

* **screen_res**: resolution of the user screen (width x height)

* **browser_res**: resolution of the user browser (width x height)

* **pixel_ratio**: pixel ratio of the browser

* **browser_connected**: when application has been launched. Uses browser time, which depends on user timezone.

---
Information recorded by `{shinylogs}`
## Session: example

![Session Example](images/shinylogs session info.PNG)
---
Information recorded by `{shinylogs}`
## Inputs

Data about inputs, by default all inputs are recorded (even those not define by developer, like with {htmlwidgets} : {DT}, {leaflet}, …)

* **sessionid**: the same ID as in session object

* **name**: the inputs inputId

* **timestamp**: timestamp when the input has changed

* **value**: the value taken by the input (can be a list in case of complex input)

* **type**: type of input (if defined)

* **binding**: binding for the input (if defined)

---
Information recorded by `{shinylogs}`
## Inputs: example


![Inputs Example](images/shinylogs inputs.PNG)

---
Information recorded by `{shinylogs}`
## Outputs

Data recorded each time an output is refreshed:

* **sessionid**: the same ID as in session object

* **name**: outputId of the output

* **timestamp**: timestamp when the output has been updated

* **type**: type of output (if defined)

* **binding**: binding for the output (if defined)

---
Information recorded by `{shinylogs}`
## Outputs: example


![Outputs Example](images/shinylogs outputs.PNG)

---
Information recorded by `{shinylogs}`
## Errors

Errors are recorded only when propagated through an output, this is the red message users see in application, info saved are:

* **sessionid**: the same ID as in session object

* **name**: outputId of the output where an error happened

* **timestamp**: timestamp of the error

* **error**: error message (if any)

* **value**: additional data for the error (generally NULL) (if defined)

---
Information recorded by `{shinylogs}`
## Errors: example


![Errors Example](images/shinylogs errors.PNG)

---
class: inverse, center, middle

# `{shiny.stats}`

Yet another way to log user data

---
# `{shiny.stats}`

Easy way for logging users’ activity **and** creating a dashboard to ingest the data. The only requirement is that you have an accessible database, which can be as simple as a local PostgreSQL database or a sqlite file.

First step is installation:

```r
devtools::install_github("Appsilon/shiny.stats")
```

---
# `{shiny.stats}` Set up


The next step is to initialize the database you want to use with shiny.stats. You should only do this once. The snippet below will connect to a sqlite database named user_stats:


```r
connection &lt;- DBI::dbConnect(RSQLite::SQLite(), dbname = "user_stats.sqlite")
DBI::dbDisconnect(connection)
```

---
# `{shiny.stats}` Set up (cont'd)

Now comes the potentially tricky part (depending on how your app’s authentication is set up): you’ll need to define a function to extract the username. The below example will extract it from the URL, from a dedicated username parameter.

```r
get_user &lt;- function(session) {
  parseQueryString(isolate(session$clientData$url_search))$username
}
```

---
# `{shiny.stats}` Server Side Logic

The magic happens in the server function where the database connection is established and user behavior is monitored:


```r
# creates user connection list and ensures required tables exist in DB
user_connection &lt;- initialize_connection(connection,
                                         username = get_user(session))

# register login
log_login(user_connection)

# register actions to watch...
log_click(user_connection, id = "apply_slider") # button
log_input(user_connection, input, input_id = "bins") # slider

# registering logout
log_logout(user_connection)
```

---
# View `{shiny.stats}` database

Open the sqlite file in your favorite database management tool

![shiny.stats view db](images/shiny.stats tableplus.png)
---
# Built-in `{shiny.stats}` dashboard

The most handy feature of `{shiny.stats}`? You can easily display users’ stats in an R Shiny dashboard. The following example code snippet creates a dashboard that allows you to monitor user adoption:


```r
library(shiny);library(RSQLite);library(shiny.stats);

# prepare credentials list to access logs:
db_credentials &lt;- list(DB_NAME = "user_stats.sqlite",
                       DB_DRIVER = "SQLite")

# define function for extracting username
get_user &lt;- function(session) {
  username &lt;- isolate(parseQueryString(session$clientData$url_search)$username)
  req(username)
  return(username)
}

# define ui and server
ui &lt;- shiny_stats_ui()
server &lt;- shiny_stats_server(get_user, db_credentials = db_credentials)
shinyApp(ui = ui, server = server,
         options = list(port = 8887, launch.browser = FALSE))
```
---
## Example `{shiny.stats}` dashboard


![shiny.stats view db](images/shiny.stats dashboard.png)
---
class: inverse, center, middle

# RSConnect Instrumentation Data

---

# Extending Connect with the Server API

The [Server API](https://docs/rstudio.com/connect/api) comes with the RSConnect product (automatically). I.E. this is NOT an API you develop with plumber, flask, etc.

Resources:
* [Documentation](https://docs/rstudio.com/connect/api)
* [Cookbook](https://docs.rstudio.com/connect/cookbook)
* [R Client](https://pkgs.rstudio.com/connectapi/) - `{connectapi}` (we'll cover this next)

### Not on CRAN (yet)

```r
remotes::install_github('rstudio/connectapi')
```


---
# Connect to the Connect Server API


```r
library(connectapi)

client &lt;- connect(
  server = 'https://connect.example.com',
  api_key = '&lt;SUPER SECRET API KEY&gt;'
)

# OR, if your server is defined by your environment variables, run:
client &lt;- connect()
```

If your role is "Publisher", you only get access to your content hosted on RSConnect! The "Admin" role gives you access to all content.


```r
users &lt;- get_users(client) # email, username, first &amp; last name, role, guid
groups &lt;- get_groups(client) # group info - guid &amp; group name
usage_shiny &lt;- get_usage_shiny(client) # content &amp; user guids, start and stop times
usage_static &lt;- get_usage_static(client) # content &amp; user guids, timestamp visited
some_content &lt;- get_content(client) # guid, content's name, title, description, etc

# get all content
all_content &lt;- get_content(client, limit = Inf)
```


---
# Example Instrumentation Data

For a shiny app:


```r
usage_shiny &lt;- get_usage_shiny(client) # content &amp; user guids, start and stop times
```

![Example Instrumentation Data](images/instrumentation_data_shiny.PNG)

---

## Example Instrumentation Dashboard

![Example Instrumentation Dashboard](images/instrumentation_flexdashboard.PNG)

[GitHub Repo](https://github.com/sol-eng/connect-usage)
---

## Example Instrumentation Shiny App

![Example Instrumentation Shiny App](images/instrumentation_data_shinyapp.PNG)

App: https://colorado.rstudio.com/rsc/usage-interactive/

[GitHub Repo](https://github.com/sol-eng/connect-usage)

---
class: inverse, center, middle
# Phew! Tool Review complete.

---
# In Review

- Did you gain a general working knowledge of 

  - bit.ly
  
  - Google Analytics

  - Matomo Analytics

  - `{shinylogs}`

  - `{shiny.stats}`
  
  - RSConnect instrumentation data

--

- I hope you feel encouraged!

--

- **Measuring** your app's usage make you a well informed developer!

---
# My Resources

* [R Shiny Google Analytics: How to add GA to shiny Apps](https://appsilon.com/r-shiny-google-analytics/) By Appsilon
* [Top 3 Tools to Monitor User Adoption in R Shiny](https://appsilon.com/monitoring-r-shiny-user-adoption/#shiny-stats) By Appsilon
* [`{shinylogs}` Documentation](https://dreamrs.github.io/shinylogs/index.html)
* [`{connectapi}` Documentation](https://awshpc22133.abc.amazon.biogen.com/IDEA/)
* [Youtube: Shiny Usage Tracking in RStudio Connect](https://www.youtube.com/watch?v=0iljqY9j64U) By Cole Arendt @ Rstudio

---
class: inverse, center, middle
Thank you!

# Questions?


    </textarea>
<style data-target="print-only">@media screen {.remark-slide-container{display:block;}.remark-slide-scaler{box-shadow:none;}}</style>
<script src="https://remarkjs.com/downloads/remark-latest.min.js"></script>
<script>var slideshow = remark.create({
"highlightLines": true,
"highlightLanguage": "R",
"countIncrementalSlides": false
});
if (window.HTMLWidgets) slideshow.on('afterShowSlide', function (slide) {
  window.dispatchEvent(new Event('resize'));
});
(function(d) {
  var s = d.createElement("style"), r = d.querySelector(".remark-slide-scaler");
  if (!r) return;
  s.type = "text/css"; s.innerHTML = "@page {size: " + r.style.width + " " + r.style.height +"; }";
  d.head.appendChild(s);
})(document);

(function(d) {
  var el = d.getElementsByClassName("remark-slides-area");
  if (!el) return;
  var slide, slides = slideshow.getSlides(), els = el[0].children;
  for (var i = 1; i < slides.length; i++) {
    slide = slides[i];
    if (slide.properties.continued === "true" || slide.properties.count === "false") {
      els[i - 1].className += ' has-continuation';
    }
  }
  var s = d.createElement("style");
  s.type = "text/css"; s.innerHTML = "@media print { .has-continuation { display: none; } }";
  d.head.appendChild(s);
})(document);
// delete the temporary CSS (for displaying all slides initially) when the user
// starts to view slides
(function() {
  var deleted = false;
  slideshow.on('beforeShowSlide', function(slide) {
    if (deleted) return;
    var sheets = document.styleSheets, node;
    for (var i = 0; i < sheets.length; i++) {
      node = sheets[i].ownerNode;
      if (node.dataset["target"] !== "print-only") continue;
      node.parentNode.removeChild(node);
    }
    deleted = true;
  });
})();
// add `data-at-shortcutkeys` attribute to <body> to resolve conflicts with JAWS
// screen reader (see PR #262)
(function(d) {
  let res = {};
  d.querySelectorAll('.remark-help-content table tr').forEach(tr => {
    const t = tr.querySelector('td:nth-child(2)').innerText;
    tr.querySelectorAll('td:first-child .key').forEach(key => {
      const k = key.innerText;
      if (/^[a-z]$/.test(k)) res[k] = t;  // must be a single letter (key)
    });
  });
  d.body.setAttribute('data-at-shortcutkeys', JSON.stringify(res));
})(document);
(function() {
  "use strict"
  // Replace <script> tags in slides area to make them executable
  var scripts = document.querySelectorAll(
    '.remark-slides-area .remark-slide-container script'
  );
  if (!scripts.length) return;
  for (var i = 0; i < scripts.length; i++) {
    var s = document.createElement('script');
    var code = document.createTextNode(scripts[i].textContent);
    s.appendChild(code);
    var scriptAttrs = scripts[i].attributes;
    for (var j = 0; j < scriptAttrs.length; j++) {
      s.setAttribute(scriptAttrs[j].name, scriptAttrs[j].value);
    }
    scripts[i].parentElement.replaceChild(s, scripts[i]);
  }
})();
(function() {
  var links = document.getElementsByTagName('a');
  for (var i = 0; i < links.length; i++) {
    if (/^(https?:)?\/\//.test(links[i].getAttribute('href'))) {
      links[i].target = '_blank';
    }
  }
})();
// adds .remark-code-has-line-highlighted class to <pre> parent elements
// of code chunks containing highlighted lines with class .remark-code-line-highlighted
(function(d) {
  const hlines = d.querySelectorAll('.remark-code-line-highlighted');
  const preParents = [];
  const findPreParent = function(line, p = 0) {
    if (p > 1) return null; // traverse up no further than grandparent
    const el = line.parentElement;
    return el.tagName === "PRE" ? el : findPreParent(el, ++p);
  };

  for (let line of hlines) {
    let pre = findPreParent(line);
    if (pre && !preParents.includes(pre)) preParents.push(pre);
  }
  preParents.forEach(p => p.classList.add("remark-code-has-line-highlighted"));
})(document);</script>

<script>
slideshow._releaseMath = function(el) {
  var i, text, code, codes = el.getElementsByTagName('code');
  for (i = 0; i < codes.length;) {
    code = codes[i];
    if (code.parentNode.tagName !== 'PRE' && code.childElementCount === 0) {
      text = code.textContent;
      if (/^\\\((.|\s)+\\\)$/.test(text) || /^\\\[(.|\s)+\\\]$/.test(text) ||
          /^\$\$(.|\s)+\$\$$/.test(text) ||
          /^\\begin\{([^}]+)\}(.|\s)+\\end\{[^}]+\}$/.test(text)) {
        code.outerHTML = code.innerHTML;  // remove <code></code>
        continue;
      }
    }
    i++;
  }
};
slideshow._releaseMath(document);
</script>
<!-- dynamically load mathjax for compatibility with self-contained -->
<script>
(function () {
  var script = document.createElement('script');
  script.type = 'text/javascript';
  script.src  = 'https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-MML-AM_CHTML';
  if (location.protocol !== 'file:' && /^https?:/.test(script.src))
    script.src  = script.src.replace(/^https?:/, '');
  document.getElementsByTagName('head')[0].appendChild(script);
})();
</script>
  </body>
</html>