index.qmd

---
title: "Data pipelines for R/Pharma"
format: 
  html:
    self-contained: true
---


# data-pipelines

## Structure

* Code: [rinpharma/data-pipelines](https://github.com/rinpharma/data-pipelines)
* Rendered run: [rinpharma.github.io/data-pipelines/](https://rinpharma.github.io/data-pipelines/)
* Output: Direct links below. Stored in gh-pages branch

## TO-DO

* [Put thought into columns](https://github.com/rinpharma/data-pipelines/issues/1)

## Rebuilding

This data (and this page) rebuild automatically based on the [CRON schedule specified in the github action](https://github.com/rinpharma/data-pipelines/blob/main/.github/workflows/pipeline.yml#L5).

Run the pipeline with `targets::tar_make()`

Check the pipeline with `targets::tar_manifest(fields = all_of("command"))`

Visualise the pipeline with `targets::tar_visnetwork()`

```{r, results = "asis", echo = FALSE, warnings = FALSE}
cat(c("```{mermaid}", targets::tar_mermaid(), "```"), sep = "\n")
```

## Output

The current files are available

* [processed_proceedings.parquet](output/processed_proceedings.parquet)
* [processed_team.parquet](output/processed_team.parquet)
* [processed_workshops.parquet](output/processed_workshops.parquet)

```{r}
arrow::read_parquet(
  "https://rinpharma.github.io/data-pipelines/output/processed_proceedings.parquet"
  ) |>
  dplyr::glimpse()
```

```{r}
arrow::read_parquet(
  "https://rinpharma.github.io/data-pipelines/output/processed_team.parquet"
  ) |>
  dplyr::glimpse()
```

```{r}
arrow::read_parquet(
  "https://rinpharma.github.io/data-pipelines/output/processed_workshops.parquet"
  ) |>
  dplyr::glimpse()
```

# Setup

Code to create gsheets token.

```{r, eval = FALSE}
# Generate credentials for gsheet access

pw_name <- gargle:::secret_pw_name("googlesheets4")
pw <- gargle:::secret_pw_gen()

added pwname_pw to usethis::edit_r_environ()

encrypt the service account token (I made via goodle dev console)
gargle:::secret_write(
  package = "googlesheets4",
  name = "rinpharma-4ac2ad6eba3b.json",
  input = "~/Downloads/rinpharma-4ac2ad6eba3b.json"
)

encrypted file is now in /inst/

file_name <- "rinpharma-4ac2ad6eba3b.json"
secret_name <- "googlesheets4"
path <- paste0("inst/secret/", file_name)
raw <- readBin(path, "raw", file.size(path))
json <- sodium::data_decrypt(
  bin = raw, key = gargle:::secret_pw_get(secret_name), 
  nonce = gargle:::secret_nonce()
  )
pass <- rawToChar(json)

gs4_auth(path = pass)
```