Updates from Way Lab specific implementations (#9)

* Initial commit * Add gsutil script and Assay.Works specifics (#1) * changes for assayworks project specifics * update shell script with bucket name * detemplatize; gsutil instructions * assay works docs context; simplify source dir * Move to .bat file and add Terraform role for bucket permissions (#2) * changes for assayworks project specifics * update shell script with bucket name * detemplatize; gsutil instructions * assay works docs context; simplify source dir * bucket.get svc-acct role binding * move gsutil work from .sh to .bat file Co-Authored-By: Faisal Alquaddoomi <[email protected]> * linting; remove shellcheck * rename gsutil file, remove .sh file * remove shellcheck Co-authored-by: Faisal Alquaddoomi <[email protected]> * Add multi-threading option for gsutil command (#3) * add multi-threaded option for gsutil * add docs for gsutil options * Adds documentation for Way Lab rclone usage (#4) * add way lab docs; reorganize utilities dir also add new repo diagram to increase role-based context understanding * linting change * Add Terraform state management resources and documentation (#5) * update existing tf to new location * add prep dir with state mgmt bucket * add hashicorp dir ignores * versions and lock file * rename to state-management dir * additional tf ignore * add state management instructions * state management bucket naming * single tf bucket var name * update lockfile * pre-commit image source image pin * tf backend comment * simplify state mgmt language Co-Authored-By: Faisal Alquaddoomi <[email protected]> * use specific version of dagger gh-action --------- Co-authored-by: Faisal Alquaddoomi <[email protected]> * Merge updates from 'gc-durbinlab-bucket/main' * rename and move files for template * update pre-commit deps * docs lint * ignore root terraform dir for pre-commit md lint * remove locks * data provider and receiver docs updates * poetry project updates * tf image updaet * image linting docs update * update readme for accurate role descriptions * generic bucket name replacements * update terraform provider versions + docs * Apply suggestions from code review Co-authored-by: Faisal Alquaddoomi <[email protected]> * add note about service account credential location * ignore the tf created svc account credential file --------- Co-authored-by: Faisal Alquaddoomi <[email protected]> Co-authored-by: Faisal Alquaddoomi <[email protected]>
CU-DBMI · Sep 19, 2023 · fb2c908 · fb2c908
1 parent b67b9c5
commit fb2c908
Show file tree

Hide file tree

Showing 27 changed files with 525 additions and 140 deletions.
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -15,9 +15,9 @@ jobs:
       DAGGER_CACHE_TO: type=gha,mode=max,scope=gc-cloud-storage-bucket
     steps:
       - name: Checkout
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
       - name: Dagger
-        uses: dagger/dagger-for-github@v3
+        uses: dagger/dagger-for-github@v4
         with:
           cmds: |
             project update

diff --git a/.gitignore b/.gitignore
@@ -202,3 +202,14 @@ cython_debug/
 # dagger ignores
 cue.mod/pkg
 cue.mod/dagger.*
+
+# data ignores
+*.json
+*.zip
+*.tar.gz
+
+# terraform ignores
+hashicorp/
+
+# tf credential ignore
+utilities/data-provider/service-account.json
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -14,28 +14,38 @@ repos:
       - id: detect-private-key
   # checking yaml formatting
   - repo: https://github.com/adrienverge/yamllint
-    rev: v1.29.0
+    rev: v1.32.0
     hooks:
       - id: yamllint
   # checking spelling
   - repo: https://github.com/codespell-project/codespell
-    rev: v2.2.2
+    rev: v2.2.5
     hooks:
       - id: codespell
         exclude: >
               (?x)^(
                   .*\.lock|.*\.csv
               )$
+  # checking markdown formatting
+  - repo: https://github.com/executablebooks/mdformat
+    rev: 0.7.17
+    hooks:
+      - id: mdformat
+        # ignore terraform autogenerated docs
+        exclude: >
+              (?x)^(
+                  terraform/.* |
+                  terraform/.*/.*
+              )$
   # linting for terraform
   - repo: https://github.com/antonbabenko/pre-commit-terraform
-    rev: v1.77.0
+    rev: v1.83.3
     hooks:
       - id: terraform_docs
         args:
           - --hook-config=--path-to-file=README.md
           - --hook-config=--add-to-existing-file=true
           - --hook-config=--create-file-if-not-exist=true
       - id: terraform_fmt
-      - id: terraform_validate
       - id: terraform_tflint
       - id: terraform_tfsec
diff --git a/README.md b/README.md
@@ -2,36 +2,89 @@
 
 ```mermaid
 flowchart LR
-    service-account -.-> files(("file(s)"))
-    files --> B[\Bucket/]
+    subgraph upload
+      files[("file(s)")]
+      data-provider["👤 Data Provider"]
+
+    end
+    subgraph download
+      data-receiver["👤 Data Receiver"]
+      files2[("file(s)")]
+    end
+    bucket[\"Cloud Storage\n Bucket"/]
+    files --> |to| bucket
+    data-provider --> |uploads| files
+    bucket --> |provides\naccess to| files2
+    files2 --> |received by| data-receiver
 
 ```
 
 Template for creating [Cloud Storage](https://cloud.google.com/storage/) bucket on [Google Cloud](https://cloud.google.com/) with a service account and related key to enable data or file upload and use.
 
 This repository uses [Terraform](https://developer.hashicorp.com/terraform/intro) to maintain cloud resources. See [terraform/README.md](terraform/README.md) for documentation on Terraform elements.
 
+## 👥 Roles
+
+See below for an overview of roles which are important to context for various parts of this repository.
+
+- __Terraform Administrator__: this role involves administrating over cloud resources created with Terraform. Content found under the `terraform` directory and following steps under [Tutorial: Bucket Infrastructure](#%EF%B8%8F-bucket-infrastructure) apply to this role.
+- __Data Provider__: this role involves using content under `utilties/data-provider` to synchronize (add, update, or remove) data to the bucket created by a Terraform Administrator. Instructions specific to this role are provided under [`utilities/data-provider/README.md`](utilities/data-provider/README.md).
+- __Data Receiver__: this role is involved with downloading content from the bucket after it has been uploaded by the data provider. Associated content may be found under [`utilities/data-receiver/README.md`](utilities/data-receiver/README.md).
+
 ## 🛠️ Install
 
 See below for steps which are required for installation.
 
 1. [Create a repository from this template](https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-repository-from-a-template).
-1. [Clone the repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository)
+
+1. [Clone the repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) to your development environment.
+
 1. Install [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)
 
+1. Configure Terraform as necessary for your Google Cloud environment.
+
 ## :books:Tutorial
 
-See below for a brief tutorial on how to implement the work found in this repository for your needs.
+See below for brief tutorials on how to implement the work found in this repository for your needs.
+
+### 🎛️ State Management
+
+These steps cover how to store [Terraform state management](https://developer.hashicorp.com/terraform/language/state) files in association with the bucket infrastructure below. Terraform tracks cloud resources it creates as a statefile (`.tfstate`). If multiple people want to manage the same resources at the same time, they all need to have access to the same statefile or else they risk overwriting or corrupting state data. One option for sharing this statefile is to use a Google Cloud Bucket, which is the option used here.
+
+Note: Terraform cloud state management must be setup before it is referenced as a backend.
 
 1. Make adjustments to the content as necessary (for example, this readme file).
-1. Fill in [terraform/variables.tf](terraform/variables.tf) with values that make sense for your initiative.
-1. Terraform __init__: to ensure Terraform is initialized use command `terraform -chdir=terraform init`.
-1. Terraform __plan__: to plan the work and observe any needs use command `terraform -chdir=terraform plan` .
-1. Terraform __apply__: to apply the work and create resources use command `terraform -chdir=terraform apply`
+1. Fill in [terraform/state-management/variables.tf](terraform/variables.tf) with values that make sense for your initiative.
+1. Terraform __init__: to ensure Terraform is initialized use command `terraform -chdir=terraform/state-management init`.
+1. Terraform __plan__: to plan the work and observe any needs use command `terraform -chdir=terraform/state-management plan` .
+1. Terraform __apply__: to apply the work and create resources use command `terraform -chdir=terraform/state-management apply`
+
+### 🏗️ Bucket Infrastructure
+
+These steps cover how to control the infrastructure found within this repository.
+
+| <span style="text-align:left;float:left;font-weight:normal;">:exclamation:  Please note: after applying the Terraform code with the steps below, a `service-account.json` file is added to your local `/utilities/data-provider` directory which contains sensitive data which may enable access to your cloud resources. __This file should not be checked into source control!__</span>   |
+|-----------------------------------------|
+
+1. Make adjustments to the content as necessary (for example, this readme file).
+1. Fill in [terraform/operations/variables.tf](terraform/variables.tf) with values that make sense for your initiative.
+1. Terraform __init__: to ensure Terraform is initialized use command `terraform -chdir=terraform/operations init`.
+1. Terraform __plan__: to plan the work and observe any needs use command `terraform -chdir=terraform/operations plan` .
+1. Terraform __apply__: to apply the work and create resources use command `terraform -chdir=terraform/operations apply`
 
 When finished with the work, optionally use the following step.
 
-- __OPTIONAL__: Terraform destroy: : to destroy all created resources use command `terraform -chdir=terraform destroy`
+- __OPTIONAL__: Terraform __destroy__: to destroy all created resources use command `terraform -chdir=terraform/operations destroy`
+
+### 📁 Using the Bucket
+
+These steps cover an example of how to use the bucket after creating the surrounding infrastructure.
+
+| <span style="text-align:left;float:left;font-weight:normal;"> ⚠️  Please note: be certain data you upload to Google Cloud abide any data governance or privacy restrictions applicable to your environment. The steps below do not inherently check or validate that data, the bucket, or the Google Cloud environment follow these policies. </span>   |
+|-----------------------------------------|
+
+- Data Upload (Data Provider): please see [`utilities/data-provider/README.md`](utilities/data-provider/README.md) for more information.
+- Data Download (Data Receiver): please see [`utilities/data-receiver/README.md`](utilities/data-receiver/README.md) for more information.
 
 ## 🧑‍💻 Development