test_system.qmd

# System tests {#sec-tests-system}

<!---
https://jakubsob.github.io/blog/

https://www.freecodecamp.org/news/clean-coding-for-beginners/
-->

```{r}
#| eval: true 
#| echo: false 
#| include: false
source("_common.R")
library(testthat)
library(gt)
```

```{r}
#| label: co_box_tldr
#| echo: false
#| results: asis
#| eval: true
co_box(
  color = "b",
  look = "default", hsize = "1.10", size = "1.05",
  header = "TLDR &emsp; ![](images/shinytest2.png){width='50' height='55'}",
  fold = TRUE,
  contents = "
<br>
  
<h5>**System tests**</h5>
  
`shinytest2` streamlines and enhances system testing for Shiny applications, making it easier to ensure your app works flawlessly inside your app-package:

- Use `record_test()` to capture user interactions and generate tests without writing code manually.
  
- Leverage `AppDriver$new()` for scripting and automating tests, making it simpler to test complex user scenarios.
  
- Apply `exportTestValues()` and `test.mode` to access and assess internal app states, offering a deeper testing perspective.
  
**BDD functions**
  
  - Use `describe()` and `it()` to add features and sceanrios in test files.\n   
  
    \`\`\`r
    describe('Feature...', code = {
        it('Scenario...', code = { ... })
    })
    \`\`\`


  "
)
```

---

This chapter covers using `shinytest2` and `testthat` to perform system tests on the features and scenarios in your app-package. 

```{r}
#| label: shinypak_apps
#| echo: false
#| results: asis
#| eval: true
shinypak_apps(regex = "^18", branch = "18_test-system")
```

> "*Failure to allow enough time for system test, in particular, is peculiarly disastrous. Since the delay comes at the end of the schedule, no one is aware of schedule trouble until almost the delivery date. Bad news, late and without warning, is unsettling to customers and to managers.*" - ['The Mythical Man-Month', Frederick P. Brooks Jr.](https://www.goodreads.com/en/book/show/13629)

System (or end-to-end) tests simulate real user interactions in a 'pre-production' environment to verify the whole application (or system) works.[^tests-prod] Approaches to system testing vary, but in general, we'll want to run a system test for each feature in our application before a release. 

If we've been documenting our unit and integration tests with BDD feature and scenario descriptions, the system tests can be used to confirm the functional requirements for the primary execution path (or user experience) before release.

[^tests-prod]: System tests should strive to replicate the production *conditions*, even when/if it's not possible to perfectly replicate the environment.

## Current tests

```{r}
#| label: git_box_17_tests-modules
#| echo: false
#| results: asis
#| eval: true
git_margin_box(
  contents = "launch",
  fig_pw = '70%', 
  branch = "17_test-modules", 
  repo = 'sap')
```

The current files in our tests folder are below: 

```{verbatim}
tests
├── testthat
│   ├── _snaps
│   │   └── text_logo.md
│   ├── fixtures
│   │   ├── make-tidy_ggp2_movies.R
│   │   └── tidy_ggp2_movies.rds
│   ├── helper.R
│   ├── setup-shinytest2.R
│   ├── test-mod_scatter_display_server.R
│   ├── test-mod_var_input_server.R
│   ├── test-scatter_plot.R
│   └── test-text_logo.R
└── testthat.R

4 directories, 10 files
```

The output from `devtools::test()` is below:

```{r}
#| eval: false 
#| code-fold: false
devtools::test()
```

```{verbatim}
✔ | F W  S  OK | Context
⠏ |          0 | mod_scatter_display_server                                               
TEST: START [2025-02-12 07:47:04] COLLECT = collected module values

TEST: END [2025-02-12 07:47:04] COLLECT = collected module values

TEST: START [2025-02-12 07:47:04] OUTPUT = is list
⠙ |          2 | mod_scatter_display_server                                               
TEST: END [2025-02-12 07:47:04] OUTPUT = is list

TEST: START [2025-02-12 07:47:04] OUTPUT = names

TEST: END [2025-02-12 07:47:04] OUTPUT = names

TEST: START [2025-02-12 07:47:04] OUTPUT = Plot object
⠸ |          4 | mod_scatter_display_server                                               
TEST: END [2025-02-12 07:47:04] OUTPUT = Plot object

TEST: START [2025-02-12 07:47:04] OUTPUT = is ggplot

TEST: END [2025-02-12 07:47:04] OUTPUT = is ggplot
✔ |          5 | mod_scatter_display_server
⠏ |          0 | mod_var_input_server                                                     
TEST: START [2025-02-12 07:47:05] RETURNED = test_vals vs. session$returned()

TEST: END [2025-02-12 07:47:05] RETURNED = test_vals vs. session$returned()
✔ |          1 | mod_var_input_server
⠏ |          0 | scatter_plot                                                             
TEST: START [2025-02-12 07:47:05] Step1 = loading movies data

TEST: END [2025-02-12 07:47:05] Step1 = movies data loaded successfully

TEST: START [2025-02-12 07:47:05] Step2 = creating graph

TEST: END [2025-02-12 07:47:05] Step2 = graph created
✔ |          1 | scatter_plot
⠏ |          0 | text_logo                                                                
TEST: START [2025-02-12 07:47:05] snap = text_logo()

TEST: END [2025-02-12 07:47:05] snap = text_logo()

TEST: START [2025-02-12 07:47:05] snap = text_logo('})')
⠙ |          2 | text_logo                                                                
TEST: START [2025-02-12 07:47:05] snap = text_logo('h1')

TEST: START [2025-02-12 07:47:05] snap = text_logo('invalid')

TEST: END [2025-02-12 07:47:05] snap = text_logo('invalid')
✔ |          3 | text_logo

══ Results ════════════════════════════════════════════════════════════════
Duration: 1.8 s

[ FAIL 0 | WARN 0 | SKIP 0 | PASS 10 ]


Nice code.
```


## Setting up [`shinytest2`]{style="font-size: 1.05em;"} {#sec-tests-system-shinytest2}

`shinytest2` requires a few steps to get up and running (most notably the [`chromote` package](https://rstudio.github.io/chromote/)), but you'll find excellent documentation on the [package website](https://rstudio.github.io/shinytest2/index.html).[^shinytest2-start] 


[^shinytest2-start]: A great place to start is the [Getting Started](https://rstudio.github.io/shinytest2/articles/shinytest2.html) vignette. 


The `shinytest2::use_shinytest2()` performs the following setup steps: 

-   ✔ Adding `shinytest2::load_app_env()` to `tests/testthat/setup-shinytest2.R`

-   ✔ Adding `*_.new.png` to `.gitignore`

-   ✔ Adding `_\\.new\\.png$` to `.Rbuildignore`

-   ✔ Adding `shinytest2` to `Suggests` field in `DESCRIPTION`

We also get some advice on using `shinytest2` functions in our code: 

```{verbatim}
#| eval: false 
#| code-fold: false
• In your package code, use `rlang::is_installed("shinytest2")` or
  `rlang::check_installed("shinytest2")` to test if shinytest2 is installed
• Then directly refer to functions with `shinytest2::fun()`
```

*After setting up `shinytest2`, be sure you can create a new chromote session like the one below:*

```{r}
#| eval: false 
#| code-fold: false
library(chromote)
b <- ChromoteSession$new()
b$view()
```

:::{#fig-tests_sys_chromote_new}

![Chromium headless browser](images/tests_sys_chromote_new.png){#fig-tests_sys_chromote_new width='100%' align='center'}

A new `Chromote` session
:::

If you run into problems running `shinytest2` tests, you can upgrade `chromium` (if you're on Mac using [homebrew](https://brew.sh/)):

```{bash}
#| eval: false 
#| code-fold: false
brew install --cask chromium
```

## Record a test {#sec-tests-system-recording-tests}

If we launch the test recorder with `shinytest2::record_test()`, change the inputs in our application, click on **Expect Shiny values** and **Save test and exit**, a test is recorded to a new test file: `tests/testthat/test-shinytest2.R`


![Creating a test with `shinytest2::record_test()`](images/tests_sys_record_test.png){width='100%' align='center'}


The test runs and saves the PNG snapshot and test values to the `tests/testthat/_snaps/` folder: 

``` sh
Loading required package: shiny
[ FAIL 0 | WARN 2 | SKIP 0 | PASS 1 ]

── Warning (test-shinytest2.R:10:3): {shinytest2} recording: sap-feature-01 ────────
Adding new file snapshot: 'tests/testthat/_snaps/sap-feature-01-001_.png'

── Warning (test-shinytest2.R:10:3): {shinytest2} recording: sap-feature-01 ────────
Adding new file snapshot: 'tests/testthat/_snaps/sap-feature-01-001.json'

[ FAIL 0 | WARN 2 | SKIP 0 | PASS 1 ]
```

```{r}
#| label: co_box_system_tests
#| echo: false
#| results: asis
#| eval: true
co_box(
  color = "r", 
  look = "default", hsize = "1.10", size = "1.05",
  fold = TRUE, 
  header = "loadSupport warning with shinytest2",
  contents = "
After setting up `shinytest2`, the `tests/testthat/setup-shinytest2.R` file contains a call to `shinytest2::load_app_env()`. This runs automatically with `shinytest2` tests and produces a familiar warning: 
  
\`\`\`bash
Warning message:
In shiny::loadSupport(app_dir, renv = renv, globalrenv = globalrenv) :
  Loading R/ subdirectory for Shiny application, but this directory appears
  to contain an R package. Sourcing files in R/ may cause unexpected behavior.
\`\`\`
  
We covered this warning message in the [Launch](launch.qmd) chapter, and it's being addressed in a future release of [`shinytest2`](https://github.com/rstudio/shinytest2/issues/308)
  
  "
)
```

If we view the contents `test-shinytest2.R` we find each action we performed in the test recorder has a corresponding call in the test:

```{r}
#| eval: false 
#| code-fold: false
library(shinytest2)
test_that("{shinytest2} recording: sap-feature-01", {
  app <- AppDriver$new(name = "sap-feature-01", height = 887, width = 1241) # <1>
  app$set_inputs(`vars-y` = "imdb_num_votes") # <2>
  app$set_inputs(`vars-x` = "critics_score") # <3>
  app$set_inputs(`vars-z` = "genre") # <4>
  app$set_inputs(`aes-alpha` = 0.7) # <5>
  app$set_inputs(`aes-size` = 3) # <6>
  app$set_inputs(`aes-plot_title` = "New plot title") # <7>
  app$expect_values() # <8>
})
```
1. Initialize the `AppDriver$new()` with the name of the test and the dimensions of the Chromium browser.  
2. Change the y axis (`vars-y`) to 'IMBD number of votes' (`imdb_num_votes`)   
3. Change the x axis (`vars-x`) to 'Critics Score' (`critics_score`)   
4. Change the color by (`vars-z`) to 'Genre' (`genre`)    
5. Change the point opacity (`aes-alpha`) to '0.7'    
6. Change the point size (`aes-alpha`) to '3'    
7. Change the plot title to (`aes-plot_title`) to 'New plot title' 
8. Export the test values 

We'll use this initial test as a template for writing the steps in our test `Scenario`s. 

## [`shinytest2`]{style="font-size: 1.10em;"} and BDD {#sec-tests-system-shinytest2-bdd}

There are multiple ways to approach your test layout with `testthat`'s `describe()`, `it()` and/or `test_that()` functions. Below is an example with dedicated `Feature` and `Scenario` descriptions, and a reference to the feature number in an `it()` call: 

```{r}
#| eval: false 
#| code-fold: false
describe("Feature 1: Scatter plot data visualization dropdowns
           As a film data analyst
           I want to explore variables in the movie review data
           So that I can analyze relationships between movie reivew sources", {
             
describe("Scenario: Change dropdown values for plotting
            Given the movie review application is loaded
            When I choose the variable [ ] for the x-axis
            And I choose the variable [ ] for the y-axis
            And I choose the variable [ ] for the color
            And I choose the size of the points to be [ ]
            And I choose the opacity of the points to be [ ]
            And I enter '[ ]' for the plot title
           
            Then the scatter plot should show [ ] on the x-axis
            And the scatter plot should show [ ] on the y-axis
            And the points on the scatter plot should be colored by [ ]
            And the size of the points on the scatter plot should be [ ]
            And the opacity of the points on the scatter plot should be [ ]
            And the title of the plot should be '[ ]'", {
              
              it("Feature 01", {
                    app <- AppDriver$new(name = "feature-01", 
                                       height = 800, width = 1173)
                  app$set_inputs(`vars-y` = "imdb_num_votes")
                  app$set_inputs(`vars-x` = "critics_score")
                  app$set_inputs(`vars-z` = "genre")
                  app$set_inputs(`aes-alpha` = 0.7)
                  app$set_inputs(`aes-size` = 3)
                  app$set_inputs(`aes-plot_title` = "New plot title")
                  app$expect_values()
              })
      })
})
```

With this approach, you can create the test file as soon as you have a `Feature` description (and come back later to fill in the `Scenario`s and tests).

An alternative approach is to use nested `describe()` functions with each `Then` step in the `it()` call (these are what will actually be tested):

```{r}
#| eval: false 
#| code-fold: false
library(shinytest2)
describe("Feature 1: Scatter plot data visualization dropdowns
           As a film data analyst
           I want to explore variables in the movie review data
           So that I can analyze relationships between movie review sources", {
  # <1>
  describe("Scenario A: Change dropdown values for plotting 
             Given the movie review application is loaded
             When I choose the variable [ ] for the x-axis
             And I choose the variable [ ] for the y-axis
             And I choose the variable [ ] for the color", {
        it("Then the scatter plot should show [ ] on the x-axis
             And the scatter plot should show [ ] on the y-axis
             And the points on the scatter plot should be colored by [ ]", {
              app <- AppDriver$new(name = "feature-01-scenario-a",
                                     height = 800, width = 1173)
                app$set_inputs(`vars-y` = "imdb_num_votes")
                app$set_inputs(`vars-x` = "critics_score")
                app$set_inputs(`vars-z` = "genre")
                app$expect_values()
       })
     }) 
   # <1>
  # <2>
  describe("Scenario B: Change dropdown values for plotting
              Given the movie review application is loaded
              When I choose the size of the points to be [ ]
              And I choose the opacity of the points to be [ ]
              And I enter '[ ]' for the plot title", {
         it("Then the size of the points on the scatter plot should be [ ]
              And the opacity of the points on the scatter plot should be [ ]
              And the title of the plot should be '[ ]'", {
              app <- AppDriver$new(name = "feature-01-scenario-b",
                                     height = 800, width = 1173)
                app$set_inputs(`aes-alpha` = 0.7)
                app$set_inputs(`aes-size` = 3)
                app$set_inputs(`aes-plot_title` = "New plot title")
                app$expect_values()
        })
      })
  # <2>
})
```
1. Test for scenario A  
2. Test for scenario B

An important note with this approach is the different names for each `AppDriver$new()` (otherwise we'd be overwriting the previous snapshot/values).

### Testing multiple apps

If we want to test a feature for one of our alternative applications in the `inst/` folder, we can use `system.file()` to pass their location to the `app_dir` argument of `AppDriver$new()`. 

In the test below, the scenario describes changing inputs for x, y, and color: 

```{r}
#| eval: false 
#| code-fold: false
library(shinytest2)
describe(
  "Feature 1: Scatter plot data visualization dropdowns
     As a film data analyst
     I want to explore movie review variables 
     So that I can analyze relationships between movie attributes and ratings", {
  describe(
    "Scenario: Change dropdown values for plotting
        Given the movie review application is loaded
        When I choose the variable ['Critics Score'] for the x-axis
        And I choose the variable ['IMDB number of votes'] for the y-axis
        And I choose the variable ['Genre'] for the color", code = {
    it("Then the scatter plot should show ['Critics Score'] on the x-axis
        And the scatter plot should show ['IMDB number of votes'] on the y-axis
        And the points on the scatter plot should be colored by ['Genre']", {
            
    test_logger(start = 'prod-feat-01', msg = "update x, y, and z")
      
    app <- AppDriver$new(system.file("prod/app", package = "sap"), # <1>
                         name = "prod-feat-01", 
                         wait = FALSE, timeout = 3000, # <2>
                         height = 800, width = 1173)
    
          app$set_inputs(`vars-y` = "imdb_num_votes")
          app$set_inputs(`vars-x` = "critics_score")
          app$set_inputs(`vars-z` = "genre")
          app$set_inputs(`aes-alpha` = 0.7)
          app$set_inputs(`aes-size` = 3)
          app$set_inputs(`aes-plot_title` = "New plot title")

          app$expect_values()
          
      test_logger(end = 'prod-feat-01', msg = "update x, y, and z")
      
      })
   })
})
```
1. Use `system.file()` to access application in `inst/tidy-data`  
2. Adjust the `wait` and `timeout` so the test will run  


Note that I've changed the `wait` and `timeout` arguments in `AppDriver$new()` because this tests takes over 10 seconds to complete (which I can see with my `test_logger()` output). When I confirm this in the output `png` file, I can see the x, y, and color values have been changed (and the missing values have been removed).

![File saved in `tests/testthat/_snaps/prod-feat-01-001_.png'`](images/prod-feat-01-001_.png){width='100%' align='center'}


## The [`test.mode`]{style="font-size: 1.05em;"} option  {#sec-tests-system-test-mode}

We've included an argument in both of the standalone app functions to allow for `options` to be passed to `shinyApp()`:

```{r}
#| eval: false 
#| code-fold: false
launch_app(options = list())
```

If we're testing our application, we can include the `test.mode = TRUE` option, which will return any values passed to `exportTestValues()`:

```{r}
#| eval: false 
#| code-fold: false
launch_app(options = list(test.mode = TRUE), run = 'p')
```

We can also include this in our `.Rprofile` as:[^cover-profile]

[^cover-profile]: We covered the `.Rprofile` in @sec-dev-rprofile.

```{r}
#| eval: false 
#| code-fold: false
options(shiny.testmode = TRUE)
```

## Exporting test values {#sec-tests-system-export-values}

To export values, place the name of exported reactive values in curly brackets (`{}`). Below is an example using the `inputs()` reactive object in the `mod_scatter_display_server()`:

```{r}
#| eval: false 
#| code-fold: false
exportTestValues(
   x = { inputs()$x },
   y = { inputs()$y },
   z = { inputs()$z },
   alpha = { inputs()$alpha },
   size = { inputs()$size },
   title = { inputs()$plot_title }
  )
```

In our test, we can create the `AppDriver$new()` object, extract the values with `get_values()`:

```{r}
#| eval: false 
#| code-fold: false
app <- AppDriver$new(name = "test-values",
                     height = 800, width = 1173,
                     wait = FALSE, timeout = 300000)

test_values <- app$get_values()

test_values[['export']]
```

```{verbatim}
$`plot-alpha`
[1] 0.5

$`plot-size`
[1] 2

$`plot-title`
[1] ""

$`plot-x`
[1] "imdb_rating"

$`plot-y`
[1] "audience_score"

$`plot-z`
[1] "mpaa_rating"
```

Now we can write tests against any of the exported values:

```{r}
#| eval: false 
#| code-fold: false
expect_equal(object = test_values[['export']]$`plot-alpha`, expected = 0.5)
expect_equal(object = test_values[['export']]$`plot-size`, expected = 2)
expect_equal(object = test_values[['export']]$`plot-title`, expected = "")
expect_equal(object = test_values[['export']]$`plot-x`, expected = "imdb_rating")
expect_equal(object = test_values[['export']]$`plot-y`, expected = "audience_score")
expect_equal(object = test_values[['export']]$`plot-z`, expected = "mpaa_rating")
```

## Test failures 

In Positron ![](images/positron.png){height=20}, the **Testing** menu item gives us a display of the test files and individual tests. 

![Tests in Positron (click to enlarge)](images/tests_system_positron_layout.png){width='100%' fig-align='center'}

When a test fails, we're alerted of the failure and the specific test the output didn't meet the expectation:

![Test failures in Positron (click to enlarge)](images/tests_system_positron_failure.png){width='100%' fig-align='center'}

In this case, we can see it's just a typo in the expected output. After making the change, we can re-run the tests and view the output in the **TEST RESULTS**:

![Fixing test failures in Positron (click to enlarge)](images/tests_system_positron_retest.png){width='100%' fig-align='center'}

**TEST RESULTS** also keeps a record of the previous test runs for us to review. 

## Test [`_snaps/`]{style="font-size: 1.05em;"}

After writing our system tests and running `devtools::test()`, the `tests/testthat/_snaps/` folder contains the follow folders and files:

```{verbatim}
tests/testthat/_snaps/
├── shinytest2
│   ├── feature-01-scenario-a-001.json
│   ├── feature-01-scenario-a-001_.png
│   ├── feature-01-senario-b-001.json
│   └── feature-01-senario-b-001_.png
├── shinytest2-prod
│   ├── prod-feat-01-001.json
│   └── prod-feat-01-001_.png
└── text_logo.md

3 directories, 7 files
```

These outputs correspond to the test files that create snapshots in the `tests/testthat/` folder:

```{verbatim}
tests/testthat/
├── test-shinytest2-export.R
├── test-shinytest2-prod.R
└── test-shinytest2.R
```

```{r}
#| label: co_box_snaps
#| echo: false
#| results: asis
#| eval: true
co_box(
  color = "g", 
  look = "default", hsize = "1.25", size = "1.05",
  header = "The `_snaps` folder",
  contents = "
I've removed the `tests/testthat/_snaps/` folder from the GitHub repo, so users can run the tests and generate the snapshots in their local environment.
")
```

 

```{r}
#| label: git_box_12e_tests-system
#| echo: false
#| results: asis
#| eval: true
git_margin_box(
  contents = "launch",
  fig_pw = '75%', 
  branch = "18_tests-system", 
  repo = 'sap')
```

## Recap {.unnumbered}

Adopting a behavior-driven development approach in system testing can help to fill the gulf between non-technical stakeholders and developers by encouraging natural, descriptive language. Using complete sentences to describe to define and communicate the application requirements will make it easier for anyone to catch unexpected behaviors.  

Using 'Features' (*As a *, *I want*, *So that*) and 'Scenarios' (*Given*, *When*, *Then*) make system tests clear (and easier to update if the specifications change). 

```{r}
#| label: co_box_app_recap
#| echo: false
#| results: asis
#| eval: true
co_box(
  color = "g",
  header = "Recap &emsp; ![](images/shinytest2.png){width='8%'}",
  fold = FALSE,
  contents = "
  
<h5>**System tests**</h5>
  
- `record_test()` simplifies the creation of system tests by recording interactions with your Shiny app. 
  - Using `record_test()` accelerates test creation and ensures our tests accurately reflect user behavior, making it easier to catch issues that could affect user experience.
  
- Use `AppDriver$new()` to manually create tests for more control and customization. 
  - Scripting tests with `shinytest2` allows for detailed user interactions and testing specific functionalities in isolation or under unique conditions, offering a granular approach to system testing.
  
- Use `exportTestValues()` in tandem with `test.mode` to expose and verify the internal state of a Shiny app during tests. 
  - This technique is crucial for testing the logic and data behind your app's UI, ensuring that the app looks right and operates correctly under various scenarios. 
  
**BDD functions**
  
  - Using `testthat`'s BDD functions makes system tests easier to update if the features and scenarios change.
  
    - Capturing the application’s desired behaviors in `Features` (_As a_ , _I want_, _So that_) and `Scenarios` (_Given_, _When_, _Then_) provides a testing script that’s clear and easy to follow.
  
  - use `describe()` and `it()` to add features and sceanrios in test files.\n   
  
    \`\`\`r
    describe('Feature...', code = {
        it('Scenario...', code = { ... })
    })
    \`\`\`

"
)
```


```{r}
#| label: git_contrib_box
#| echo: false
#| results: asis
#| eval: true
git_contrib_box()
```