Devcontainer support including Duckdb bootstrapped with minimal synthetic data

.devcontainer/devcontainer.json

@@ -0,0 +1,44 @@
+// For format details, see https://aka.ms/devcontainer.json. For config options, see the
+// README at: https://github.com/devcontainers/templates/tree/main/src/python
+{
+    "name": "Python 3",
+    // Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
+    "image": "mcr.microsoft.com/devcontainers/python:3.12",
+    // Features to add to the dev container. More info: https://containers.dev/features.
+    // "features": {},
+    // Use 'forwardPorts' to make a list of ports inside the container available locally.
+    // "forwardPorts": [],
+    // Use 'postCreateCommand' to run commands after the container is created.
+    "postCreateCommand": "bash ./.devcontainer/scripts/postCreate.sh",
+    "postStartCommand": "bash ./.devcontainer/scripts/postStart.sh",
+    // "mounts": [],
+    "customizations": {
+        "vscode": {
+            "extensions": [
+                "davidanson.vscode-markdownlint",
+                "editorconfig.editorconfig",
+                "mads-hartmann.bash-ide-vscode",
+                "mechatroner.rainbow-csv",
+                "ms-python.black-formatter",
+                "ms-python.python",
+                "ms-python.vscode-pylance",
+                "njpwerner.autodocstring",
+                "innoverio.vscode-dbt-power-user"
+            ],
+            "settings": {
+                "python.formatting.provider": "black",
+                "python.analysis.completeFunctionParens": true,
+                "[python]": {
+                    "editor.defaultFormatter": "ms-python.black-formatter",
+                    "editor.formatOnSave": true
+                },
+                "python.defaultInterpreterPath": "./dbt-env/bin/python3",
+                "dbt.dbtPythonPathOverride": "./dbt-env/bin/python3"
+            }
+        }
+    },
+    // Configure tool-specific properties.
+    // "customizations": {},
+    // Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
+    // "remoteUser": "root"
+}

.devcontainer/scripts/minimal_requirements.txt

@@ -0,0 +1,5 @@
+dbt-core==1.8.2
+dbt-duckdb==1.8.1
+pre-commit==3.7.1
+black
+python-dotenv

.devcontainer/scripts/postCreate.sh

@@ -0,0 +1,40 @@
+# Set up git
+git config --global --add safe.directory /workspaces/dbt-synthea
+git config --global init.defaultBranch main
+
+# Install requirements
+cd /workspaces/dbt-synthea
+python -m venv dbt-env
+source dbt-env/bin/activate 
+pip install -r .devcontainer/scripts/minimal_requirements.txt
+
+# Setup pre-commit
+pre-commit install
+
+# Setup bash history search
+cat >> ~/.inputrc <<'EOF'
+"\e[A": history-search-backward
+"\e[B": history-search-forward
+EOF
+
+
+
+# Setup dbt profile
+mkdir /home/vscode/.dbt
+cat >> /home/vscode/.dbt/profiles.yml <<'EOF'
+synthea_omop_etl:
+  target: dev
+  outputs:
+    dev:
+      type: duckdb
+      path: ./data/synthea_omop_etl.duckdb
+      schema: dbt_synthea_dev
+EOF
+
+# Setup dbt
+
+echo "Setting up duckdb synthetic data and dbt"
+mkdir ./data
+
+dbt deps
+dbt seed

.devcontainer/scripts/postStart.sh

@@ -0,0 +1,2 @@
+git config --global --add safe.directory /workspaces/dbt-synthea
+

.vscode/settings.json

@@ -5,6 +5,7 @@
     },
     "python.terminal.activateEnvInCurrentTerminal": true,
     "python.defaultInterpreterPath": "./dbt-env/bin/python3",
+    "dbt.dbtPythonPathOverride": "./dbt-env/bin/python3",
     "dbt.queryLimit": 500,
     "yaml.schemas": {
         "https://raw.githubusercontent.com/dbt-labs/dbt-jsonschema/main/schemas/dbt_yml_files.json": [
@@ -25,4 +26,4 @@
             "packages.yml"
         ]
     },
-}
+}

README.md

@@ -1,5 +1,8 @@
+[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/OHDSI/dbt-synthea)
+
 # [Under developement] dbt-synthea
-The purpose of this project is to re-create the Synthea-->OMOP ETL implemented in https://github.com/OHDSI/ETL-Synthea using [dbt](https://github.com/dbt-labs/dbt-core).
+
+The purpose of this project is to re-create the Synthea-->OMOP ETL implemented in <https://github.com/OHDSI/ETL-Synthea> using [dbt](https://github.com/dbt-labs/dbt-core).
 
 The project is currently under development and is not yet ready for production use.
 
@@ -9,6 +12,11 @@ We built dbt-synthea to demonstrate the power of dbt for building OMOP ETLs.  **
 
 ...and this is just the beginning!  We hope someday to grow the OHDSI dbt ecosystem to include a generic dbt project template and source-specific macros and models.  Stay tuned and please reach out if you are interested in contributing.
 
+## Devcontainer development
+
+The project supports quick experimentation for developers wishing to test the entire dbt workflow on synthetic data using [DuckDb](https://duckdb.org/) through the use of a [dev container](https://containers.dev/).
+This also allows the use of a consistent, pre-configured development environment that can also be used in [GitHub Codespaces](https://github.com/features/codespaces).
+
 ## Developer Setup
 
 Currently this project is set up to run an OMOP ETL into either duckdb or Postgres.  Setup instructions for each are provided below.
@@ -18,35 +26,46 @@ By default, the project will source the Synthea and OMOP vocabulary data from se
 Users are welcomed, however, to utilize their own Synthea and/or OMOP vocabulary tables as sources.  Instructions for the "BYO data" setup are provided below.
 
 ### Prerequisites
+
 - See the top of [this page](https://docs.getdbt.com/docs/core/pip-install) for OS & Python requirements.  (Do NOT install dbt yet - see below for project installation and setup.)
 - It is recommended to use [VS Code](https://code.visualstudio.com/) as your IDE for developing this project.  Install the `dbt Power User` extension in VS Code to enjoy a plethora of useful features that make dbt development easier
 - This project currently only supports **Synthea v3.0.0**
 
 ### Repo Setup
+
  1. Clone this repository to your machine
  2. `cd` into the repo directory and set up a virtual environment:
+
  ```bash
  python3 -m venv dbt-env
  ```
- - If you are using VS Code, create a .env file in  the root of your repo workspace (`touch .env`) and add a PYTHONPATH entry for your virtual env (for example, if you cloned your repo in your computer's home directory, the entry will read as: `PYTHONPATH="~/dbt-synthea/dbt-env/bin/python"`)
- - Now, in VS Code, once you set this virtualenv as your preferred interpreter for the project, the vscode config in the repo will automatically source this env each time you open a new terminal in the project.  Otherwise, each time you open a new terminal to use dbt for this project, run:
+
+- If you are using VS Code, create a .env file in  the root of your repo workspace (`touch .env`) and add a PYTHONPATH entry for your virtual env (for example, if you cloned your repo in your computer's home directory, the entry will read as: `PYTHONPATH="~/dbt-synthea/dbt-env/bin/python"`)
+- Now, in VS Code, once you set this virtualenv as your preferred interpreter for the project, the vscode config in the repo will automatically source this env each time you open a new terminal in the project.  Otherwise, each time you open a new terminal to use dbt for this project, run:
+
 ```bash
 source dbt-env/bin/activate         # activate the environment for Mac and Linux OR
 dbt-env\Scripts\activate            # activate the environment for Windows
 ```
+
  4. In your virtual environment, install dbt and other required dependencies as follows:
+
 ```bash
 pip3 install -r requirements.txt
 pre-commit install
 ```
-   - This will install dbt-core, the dbt duckdb and postgres adapters, SQLFluff (a SQL linter),  pre-commit (in order to run SQLFluff on all newly-committed code in this repo), duckdb (to support bootstrapping scripts), and various dependencies for the listed packages
+
+- This will install dbt-core, the dbt duckdb and postgres adapters, SQLFluff (a SQL linter),  pre-commit (in order to run SQLFluff on all newly-committed code in this repo), duckdb (to support bootstrapping scripts), and various dependencies for the listed packages
 
 ### DuckDB Setup
+
  1. Create a duckdb database in this repo's `data` directory (e.g. `data/synthea_omop_etl.duckdb`)
 
  2. Set up your [profiles.yml file](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml):
-   - Create a directory `.dbt` in your root directory if one doesn't exist already, then create a `profiles.yml` file in `.dbt` 
-   - Add the following block to the file:
+
+- Create a directory `.dbt` in your root directory if one doesn't exist already, then create a `profiles.yml` file in `.dbt`
+- Add the following block to the file:
+
 ```yaml
   synthea_omop_etl:
   outputs:
@@ -58,22 +77,27 @@ pre-commit install
 ```
 
  3. Ensure your profile is setup correctly using dbt debug:
+
 ```bash
 dbt debug
 ```
 
  4. Load dbt dependencies:
+
 ```bash
 dbt deps
 ```
 
  5. **If you'd like to run the default ETL using the pre-seeded Synthea dataset,** run `dbt seed` to load the CSVs with the Synthea dataset and vocabulary data. This materializes the seed CSVs as tables in your target schema (vocab) and a _synthea schema (Synthea tables).  **Then, skip to step 9 below.**
+
 ```bash
 dbt seed
 ```
+
  6. **If you'd like to run the ETL on your own Synthea dataset,** first toggle the `seed_source` variable in `dbt_project.yml` to `false`. This will tell dbt not to look for the source data in the seed schemas.
- 
+
  7. **[BYO DATA ONLY]** Load your Synthea and Vocabulary data into the database by running the following commands (modify the commands as needed to specify the path to the folder storing the Synthea and vocabulary csv files, respectively).  The vocabulary tables will be created in the target schema specified in your profiles.yml for the profile you are targeting.  The Synthea tables will be created in a schema named "<target schema>_synthea".  **NOTE only Synthea v3.0.0 is supported at this time.**
+
 ``` bash
 file_dict=$(python3 scripts/python/get_csv_filepaths.py path/to/synthea/csvs)
 dbt run-operation load_data_duckdb --args "{file_dict: $file_dict, vocab_tables: false}"
@@ -82,26 +106,32 @@ dbt run-operation load_data_duckdb --args "{file_dict: $file_dict, vocab_tables:
 ```
 
  8. Seed the location mapper and currently unused empty OMOP tables:
+
 ```bash
 dbt seed --select states omop
 ```
 
  9. Build the OMOP tables:
+
 ```bash
 dbt run
 ```
 
  10. Run tests:
+
 ```bash
 dbt test
 ```
 
 ### Postgres Setup
+
  1. Set up a local Postgres database with a dedicated schema for developing this project (e.g. `dbt_synthea_dev`)
 
  2. Set up your [profiles.yml file](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml):
-   - Create a directory `.dbt` in your root directory if one doesn't exist already, then create a `profiles.yml` file in `.dbt` 
-   - Add the following block to the file:
+
+- Create a directory `.dbt` in your root directory if one doesn't exist already, then create a `profiles.yml` file in `.dbt`
+- Add the following block to the file:
+
 ```yaml
   synthea_omop_etl:
   outputs:
@@ -118,23 +148,27 @@ dbt test
 ```
 
  3. Ensure your profile is setup correctly using dbt debug:
+
 ```bash
 dbt debug
 ```
 
  4. Load dbt dependencies:
+
 ```bash
 dbt deps
 ```
 
  5. **If you'd like to run the default ETL using the pre-seeded Synthea dataset,** run `dbt seed` to load the CSVs with the Synthea dataset and vocabulary data. This materializes the seed CSVs as tables in your target schema (vocab) and a _synthea schema (Synthea tables).  **Then, skip to step 10 below.**
+
 ```bash
 dbt seed
 ```
- 
+
  6. **If you'd like to run the ETL on your own Synthea dataset,** first toggle the `seed_source` variable in `dbt_project.yml` to `false`. This will tell dbt not to look for the source data in the seed schemas.
- 
+
  7. **[BYO DATA ONLY]** Create the empty vocabulary and Synthea tables by running the following commands.  The vocabulary tables will be created in the target schema specified in your profiles.yml for the profile you are targeting.  The Synthea tables will be created in a schema named "<target schema>_synthea".
+
 ``` bash
 dbt run-operation create_vocab_tables
 dbt run-operation create_synthea_tables
@@ -143,16 +177,19 @@ dbt run-operation create_synthea_tables
  8. **[BYO DATA ONLY]** Use the technology/package of your choice to load the OMOP vocabulary and raw Synthea files into these newly-created tables. **NOTE only Synthea v3.0.0 is supported at this time.**
 
  9. Seed the location mapper and currently unused empty OMOP tables:
+
 ```bash
 dbt seed --select states omop
 ```
 
  10. Build the OMOP tables:
+
 ```bash
 dbt run
 ```
 
  11. Run tests:
+
 ```bash
 dbt test
-```
+```

package-lock.yml

@@ -1,4 +1,4 @@
 packages:
   - package: dbt-labs/dbt_utils
-    version: 1.2.0
-sha1_hash: d4f259856543b0ef301e0b3b0bbc94ccb6b12a54
+    version: 1.3.0
+sha1_hash: 226ae69cdfbc9367e2aa2c472b01f99dbce11de0