Marketplace deploy

.gitignore

@@ -8,6 +8,7 @@ terraform.tfstate*
 *.tfstate
 .venv
 node_modules
+looker.ini
 
 .vertex_cf_auth_token
 dist

CHANGELOG.md

@@ -1,4 +1,29 @@
 # Changelog
+
+
+## V4.1 (Marketplace PR)
+- Instead of using sql runner queries, the required data is modeled in LookML. This prevents the need for escalated privileges for users (ie use_sql_runner permission). 
+  - Requires the use of a Looker block or remote project import.
+- Environment variables previously built into the Frontend application are now entered in an admin menu of the UI and stored as user attributes. 
+  - The compiled .js is now portable.
+- Instead of calling a cloud function endpoint through the browser, this call is proxied through Looker. 
+  - Allows a closed network design.
+- Backend install flow available in Cloud Console (GCP Cloud Shell). 
+  - Local IDE setup is now unnecessary.
+- Cloud console flow for backend setup includes step by step instructions as well as Terraform script. 
+  - This makes installation in an existing Google Project possible.
+
+
+## V4.0 (Making it smarter)
+- Updated Readme to provide clearer instructions and troubleshooting tips
+- Simplified setup with a single shared environment file
+- Improved bash scripts for example generation
+- Added the ability to generate training prompts from trusted dashboards
+- Improved error messages (added CORS headers) to improve the setup process
+- Improved the accuracy of Explore Assistant returning the correct results by adding relevant context. Specific improvements include:
+  - Date filters: Accuracy for basic, range, and complex date filters increased significantly (details in table below).
+  - Field selection: Accuracy for selecting fields with pivots and dates has also improved.
+
 ## v3.1
 
 ### Added

README.md

@@ -19,6 +19,14 @@ Additionally, the extension provides:
  - Insight Summarization
  - Dynamic Explore Selection
 
+## Setup
+
+Please follow these steps in order for a successful installation.
+1. Backend Setup - setup the GCP backend for communicating with the Vertex API [using these instructions.](./explore-assistant-backend/README.md)
+2. Looker Connection - setup a Looker connection to the BigQuery dataset created in step 1 [using these instructions.](https://cloud.google.com/looker/docs/db-config-google-bigquery)
+3. Example generation - generate a list of examples and upload them to BigQuery [using these instructions.](./explore-assistant-examples/README.md)
+4. Frontend Setup - setup Looker Extension Framework Applications [using these instructions.](./explore-assistant-extension/README.md)
+
 ### Technologies Used
 #### Frontend
 - [React](https://reactjs.org/)
@@ -36,50 +44,6 @@ Additionally, the extension provides:
 - [Vertex AI](https://cloud.google.com/vertex-ai)
 - [Cloud Functions](https://cloud.google.com/functions)
 
-## Get Started
-
-Getting started involves (*in this order*):
-1. Clone or download a copy of this repository to your development machine.
-   If you have a git ssh_config:
-   ```bash
-   # cd ~/ Optional. your user directory is usually a good place to git clone to.
-   git clone [email protected]:looker-open-source/looker-explore-assistant.git
-   ```
-
-   If not:
-   ```bash
-   # cd ~/ Optional. your user directory is usually a good place to git clone to.
-   git clone https://github.com/looker-open-source/looker-explore-assistant.git
-   ```
-   Alternatively, open up this repository in:  
-   [![Open in Cloud Shell](https://gstatic.com/cloudssh/images/open-btn.svg)](https://shell.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https://github.com/looker-open-source/looker-explore-assistant.git&cloudshell_workspace=explore-assistant-extension)
-2. Make sure [pip](https://pip.pypa.io/en/stable/cli/pip_install/) is installed on your computer to run the `pip install -r requirements.txt` command line in the setup section.     
-3. Install [`google-cloud-sdk`](https://cloud.google.com/sdk/docs/install) in the looker-explore-assistant directory to install Google Cloud SDK before the backend setup. 
-        >To install google-cloud-sdk, you can use this command `brew install —cask google-cloud-sdk`. Ensure you have [Homebrew](https://brew.sh/) installed first
-4. Create a GCP Project (you’ll need the ID later). It does not have to be the same project as the prompt tables but it is recommended for simplicity
-5. Create a Looker connection for that BigQuery project
-6. Create an empty Looker project
-       - Add the connection name to the model file
-       - Configure git
-       - That’s all you need to do for now. This is where the extension framework will be deployed. The connection should be the same as the one that holds the prompts
-
-The local cloud function backend and example generation require some python packages. It is recommended to create a python virtual environment and install the dependencies:
-
-```bash
-# Use python3 on Mac OS
-python -m venv .venv
-source .venv/bin/activate 
-pip install -r ./explore-assistant-examples/requirements.txt
-pip install -r ./explore-assistant-cloud-function/requirements.txt 
-```
-> If you hit a blocker with directory permissions, use `chmod +x <FILE NAME>` to allow write permissions.
-
-## Setup
-
-1. Backend Setup - setup the GCP backend for communicating with the Vertex API [using these instructions.](./explore-assistant-backend/README.md)
-2. Example generation - generate a list of examples and upload them to BigQuery [using these instructions.](./explore-assistant-examples/README.md)
-3. Frontend Setup - setup Looker Extension Framework Applications by following [these instructions](./explore-assistant-extension/README.md).
-
 ## Recommendations for fine tuning the model
 
 This app uses a one shot prompt technique for fine tuning a model, meaning that all the metadata for the model is contained in the prompt. It's a good technique for a small dataset, but for a larger dataset, you may want to use a more traditional fine tuning approach. This is a simple implementation, but you can also use a more sophisticated approach that involves generating embeddings for explore metadata and leveraging a vector database for indexing.

explore-assistant-backend/README.md

@@ -6,16 +6,32 @@ This Terraform configuration establishes a backend for the Looker Explore Assist
 
 The Explore Assistant also uses a set of examples to improve the quality of its answers. We store those examples in BigQuery. Please see the comparisons below when deciding which deployment approach to use.
 
+## Google Project
+
+A Google Cloud Project is necessary to provision backend resources. A new google project simplifies installation. If an existing google project is used, Terraform will not be usable.
+
+## Cloud Shell Setup
+
+To simplify the backend installation, you can use the following link to open a Google Cloud Shell. 
+
+  [![Open in Cloud Shell](https://gstatic.com/cloudssh/images/open-btn.svg)](https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https://github.com/bytecodeio/looker-explore-assistant&cloudshell_tutorial=./explore-assistant-backend/cloudshell_README.md&shellonly=true&cloudshell_git_branch=marketplace_deploy)
+Within the cloud shell, these [installation instructions](./cloudshell_README.md) will be shown.
+
+## Development Setup
+
+Alternately, follow the below directions for a manual compilation and install.
+
 ### What backend should I use?
 
-Here we list the reasons and tradeoffs of each deployment approach in an effort to scope the right backend deployment approach based on individual preferences and existing setups. 
+Here we list the reasons and tradeoffs of each deployment approach in an effort to scope the right backend deployment approach based on individual preferences and existing setups. The backend setup will default for a Cloud Run installation.
 
 **Regardless of Backend**:
 * Any Looker database connection can be used for fetching the actual data returned from the natural language query url
 * They implement the same API, as in no Looker Credentials are stored in the backends and the arguments are the same (*ie. model parameters and a prompt*)
 * By default both approaches fetch examples from a BigQuery table out of simplicity. For Cloud Functions you can modify [this React Hook](../explore-assistant-extension/src/hooks/useBigQueryExamples.ts) and change the `connection_name` on line 18 to point to the non BQ database connection in Looker that houses your example prompts/training data.
 
 **For Cloud Function/Run**:
+* Default method. 
 * Generally speaking, this approach is recommended for folks who want more development control on the backend
 * Your programming language of choice can be used
 * Workflows for custom codeflow like using custom models, combining models to improve results, fetching from external datastores, etc. are supported
@@ -32,7 +48,6 @@ Here we list the reasons and tradeoffs of each deployment approach in an effort
 
 ## Prerequisites
 
-- Terraform installed on your machine.
 - Access to a GCP account with permission to create and manage resources.
 - A GCP project where the resources will be deployed.
 

explore-assistant-backend/cloudshell_README.md

@@ -0,0 +1,236 @@
+# Looker Explore Assistant Backend Service
+
+This is an automatic installer of the GCP Cloud Run backend service.
+This is intended to be installed in an empty google project. If you are using a project that already has resources in it, see the step-by-step configuration section below.
+To begin, please execute:
+```
+cd explore-assistant-backend/terraform && ./init.sh
+```
+
+## Caution
+If resources will be deleted or destroyed by Terraform, please abort the process to avoid destroying existing resources. Use the step-by-step configuration section below instead. For more information on the choice of backend, please see [the Backend README.](./README.md)
+
+# Step by Step Configuration
+
+## 1: Set up Environment Variables
+Make sure to replace PROJECT_ID and REGION values with actual values. The other values can be left as default.
+``` bash
+export PROJECT_ID="your-project-id"
+export REGION="us-central1"
+export DATASET_ID="explore_assistant"
+export CLOUD_RUN_SERVICE_NAME="explore-assistant-api"
+export VERTEX_CF_AUTH_TOKEN=$(openssl rand -base64 32)
+gcloud config set project $PROJECT_ID
+echo $VERTEX_CF_AUTH_TOKEN
+```
+Please copy the Vertex CF Auth Token that is printed out. This will be used later in the frontend setup.
+
+## 2: Enable Required APIs
+(for Both backend types)
+``` bash
+gcloud services enable serviceusage.googleapis.com \
+    cloudresourcemanager.googleapis.com \
+    iam.googleapis.com \
+    aiplatform.googleapis.com \
+    bigquery.googleapis.com \
+    cloudapis.googleapis.com \
+    cloudbuild.googleapis.com \
+    cloudfunctions.googleapis.com \
+    run.googleapis.com \
+    storage-api.googleapis.com \
+    storage.googleapis.com \
+    compute.googleapis.com \
+    secretmanager.googleapis.com \
+    aiplatform.googleapis.com
+```
+THEN, you must wait a bit. These APIs need time to activate. Have a coffee, drink some water, then come back and proceed with the next step. Let the scripts do the heavy lifting, but give them time.
+
+## 3: Create Service Account
+Did you remember to take a break before starting this step? Please do, the APIs need time to activate & propogate change to the regionless services like IAM. :coffee:
+(for Both backend types)
+```bash 
+gcloud iam service-accounts create explore-assistant-cf-sa \
+    --display-name "Looker Explore Assistant Cloud Function SA"
+
+gcloud projects add-iam-policy-binding $PROJECT_ID \
+    --member "serviceAccount:explore-assistant-cf-sa@$PROJECT_ID.iam.gserviceaccount.com" \
+    --role "roles/aiplatform.user"
+```
+
+## 4: Create Secret in Secret Manager 
+(for Cloud Function backend ONLY)
+``` bash
+echo -n $VERTEX_CF_AUTH_TOKEN | gcloud secrets create VERTEX_CF_AUTH_TOKEN \
+    --replication-policy=user-managed \
+    --locations=$REGION \
+    --data-file=-
+
+gcloud secrets add-iam-policy-binding VERTEX_CF_AUTH_TOKEN \
+    --member "serviceAccount:explore-assistant-cf-sa@$PROJECT_ID.iam.gserviceaccount.com" \
+    --role "roles/secretmanager.secretAccessor"
+```
+
+## 5: Create Storage Bucket for Cloud Function Source 
+(for Cloud Function backend ONLY)
+``` bash
+BUCKET_NAME="${PROJECT_ID}-gcf-source-$(openssl rand -hex 4)"
+gsutil mb -p $PROJECT_ID -l US gs://$BUCKET_NAME/
+```
+
+## 6: Upload Cloud Function Source Code
+(for Cloud Function backend ONLY)
+```bash
+cd ./explore-assistant-cloud-function && zip -r ../function-source.zip * && cd ..
+gsutil cp function-source.zip gs://$BUCKET_NAME/
+```
+
+## 7: Create Artifact Registry Repository
+(for Cloud Function backend ONLY)
+```bash
+gcloud artifacts repositories create explore-assistant-repo \
+    --repository-format=docker \
+    --location=$REGION \
+    --project=$PROJECT_ID \
+    --description="Docker repository for Explore Assistant"
+```
+
+## 8: Deploy Cloud Function
+(for Cloud Function backend ONLY)
+```bash
+gcloud functions deploy $CLOUD_RUN_SERVICE_NAME \
+    --gen2 \
+    --region=$REGION \
+    --project=$PROJECT_ID \
+    --runtime=python310 \
+    --entry-point=cloud_function_entrypoint \
+    --trigger-http \
+    --source=gs://$BUCKET_NAME/function-source.zip \
+    --set-env-vars=REGION=$REGION,PROJECT=$PROJECT_ID \
+    --set-secrets=VERTEX_CF_AUTH_TOKEN=VERTEX_CF_AUTH_TOKEN:latest \
+    --max-instances=10 \
+    --memory=4Gi \
+    --timeout=60s \
+    --service-account=explore-assistant-cf-sa@$PROJECT_ID.iam.gserviceaccount.com
+```
+
+## 9: Make Cloud Function Public
+(for Cloud Function backend ONLY)
+```bash
+gcloud functions add-iam-policy-binding $CLOUD_RUN_SERVICE_NAME \
+    --region=$REGION \
+    --project=$PROJECT_ID \
+    --member="allUsers" \
+    --role="roles/cloudfunctions.invoker"
+gcloud functions describe $CLOUD_RUN_SERVICE_NAME --region=$REGION --format='value(httpsTrigger.url)'
+```
+The Cloud function url will be printed out and should be copied. This will be used in the frontend installation.
+
+## 10: Create BigQuery Dataset
+(for Both backend types)
+``` bash
+bq --location=$REGION mk --dataset $PROJECT_ID:$DATASET_ID
+```
+
+## 11: Create BigQuery Tables
+(for Both backend types)
+``` bash
+bq query --use_legacy_sql=false --location=$REGION \
+    "CREATE OR REPLACE TABLE \`${DATASET_ID}.explore_assistant_examples\` (
+        explore_id STRING OPTIONS (description = 'Explore id of the explore to pull examples for in a format of -> lookml_model:lookml_explore'),
+        examples STRING OPTIONS (description = 'Examples for Explore Assistant training. JSON document with list hashes each with input and output keys.')
+    )"
+
+bq query --use_legacy_sql=false --location=$REGION \
+    "CREATE OR REPLACE TABLE \`${DATASET_ID}.explore_assistant_refinement_examples\` (
+        explore_id STRING OPTIONS (description = 'Explore id of the explore to pull examples for in a format of -> lookml_model:lookml_explore'),
+        examples STRING OPTIONS (description = 'Examples for Explore Assistant training. JSON document with list hashes each with input and output keys.')
+    )"
+
+bq query --use_legacy_sql=false --location=$REGION \
+    "CREATE OR REPLACE TABLE \`${DATASET_ID}.explore_assistant_samples\` (
+        explore_id STRING OPTIONS (description = 'Explore id of the explore to pull examples for in a format of -> lookml_model:lookml_explore'),
+        samples STRING OPTIONS (description = 'Samples for Explore Assistant Samples displayed in UI. JSON document with listed samples with category, prompt and color keys.')
+    )"
+```
+
+## 12: Create BigQuery Connection and Model  
+(For BigQuery backend install ONLY)
+> **Note:** This process is very expensive. It is better to use the Cloud Function Backend
+``` bash
+gcloud services enable bigqueryconnection.googleapis.com
+
+bq mk --connection \
+    --connection_type=CLOUD_RESOURCE \
+    --project_id=$PROJECT_ID \
+    --location=$REGION \
+    explore_assistant_llm
+
+gcloud projects add-iam-policy-binding $PROJECT_ID \
+    --member "serviceAccount:$(bq show --format=json --connection --project_id=$PROJECT_ID --location=$REGION explore_assistant_llm | jq -r .cloudResource.serviceAccountId)" \
+    --role "roles/aiplatform.user"
+
+bq query --use_legacy_sql=false --location=$REGION \
+    "CREATE OR REPLACE MODEL \`${DATASET_ID}.explore_assistant_llm\` 
+    REMOTE WITH CONNECTION \`$PROJECT_ID.$REGION.explore_assistant_llm\` 
+    OPTIONS (endpoint = 'gemini-1.5-flash')"
+```
+
+## 13: Optional: Configure Security Settings
+If you want to restrict access to your Cloud Function to Looker's specific IP ranges, you can run these additional steps. 
+First, determine the list of your [Looker IPs](https://cloud.google.com/looker/docs/enabling-secure-db-access#:~:text=The%20list%20of%20IP%20addresses,(es)%20that%20are%20shown.)
+
+## 13.1: Set variables
+Please modify the below command to include your [Looker IPs](https://cloud.google.com/looker/docs/enabling-secure-db-access#:~:text=The%20list%20of%20IP%20addresses,(es)%20that%20are%20shown.).
+```bash
+export ALLOWED_IP_ADDRESSES="your.ip.address/32,second.ip.address/32,third.ip.address/32"
+export VPC_NETWORK_NAME=explore-assistant-vpc
+export SUBNET_NAME=explore-assistant-subnet
+export VPC_CONNECTOR_NAME=eavpcconnector
+export SECURITY_POLICY_NAME=eapolicy
+```
+## 13.2: Create network and subnet
+```bash
+gcloud compute networks create $VPC_NETWORK_NAME \
+    --subnet-mode=custom
+
+gcloud compute networks subnets create $SUBNET_NAME \
+    --network=$VPC_NETWORK_NAME \
+    --region=$REGION \
+    --range=10.0.0.0/24
+```
+## 13.3 Create VPC Connector (takes a while)
+```bash
+gcloud compute networks vpc-access connectors create $VPC_CONNECTOR_NAME \
+    --network $VPC_NETWORK_NAME \
+    --region $REGION \
+    --range 10.8.0.0/28
+```
+## 13.4 Update Cloud Function to use VPC Connector
+```bash
+gcloud run services update $CLOUD_RUN_SERVICE_NAME \
+    --vpc-connector $VPC_CONNECTOR_NAME \
+    --region $REGION \
+    --project $PROJECT_ID
+```
+
+## 13.5 Create and configure security policy
+```bash
+gcloud compute security-policies create $SECURITY_POLICY_NAME \
+    --description "Restrict access to specific IP addresses"
+
+gcloud compute security-policies rules create 1000 \
+    --security-policy $SECURITY_POLICY_NAME \
+    --description "Allow Looker IP addresses" \
+    --src-ip-ranges $ALLOWED_IP_ADDRESSES \
+    --action allow
+
+gcloud compute security-policies rules create 2000 \
+    --security-policy $SECURITY_POLICY_NAME \
+    --description "Deny all other IP addresses" \
+    --src-ip-ranges="0.0.0.0/0" \
+    --action deny-403
+```
+
+## 13.6: Done with optional security restrictions
+
+For step-by-step debugging or manual configuration of the security settings, refer to the Google Cloud documentation on [Cloud Armor](https://cloud.google.com/armor/docs) and [VPC Service Controls](https://cloud.google.com/vpc-service-controls/docs).

explore-assistant-backend/terraform/backends/backend-gcs.tf

@@ -1,5 +1,6 @@
 terraform {
   backend "gcs" {
+    bucket = "project-id-terraform-state"
     prefix  = "terraform/state"
   }
-}
+}