Skip to content

Commit

Permalink
Merge remote-tracking branch 'refs/remotes/origin/main'
Browse files Browse the repository at this point in the history
  • Loading branch information
@moficodes
moficodes committed May 17, 2024
2 parents 97598ac + 70fe82b commit b6ddc5a
Show file tree
Hide file tree
Showing 4,886 changed files with 46,605 additions and 1,505,434 deletions.
The diff you're trying to view is too large. We only load the first 3000 changed files.
23 changes: 19 additions & 4 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,25 @@
# See the License for the specific language governing permissions and
# limitations under the License.

terraform.tfvars
terraform.tfstate*
.terraform*
__pycache__/
## Archives
**/*.tar
**/*.tar.gz
**/*.zip

# Directories
bin/
deploy/

# IDEs
.idea/

# Python
__pycache__/

# Terraform
default.tfstate
default.tfstate.backup
.terraform*
terraform.tfstate*
terraform.tfvars
tfplan
36 changes: 18 additions & 18 deletions applications/jupyter/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ This module deploys the following resources, once per user:
- iap.googleapis.com (required when using authentication with Identity Aware Proxy)

2. A functional GKE cluster.
- To create a new standard or autopilot cluster, follow the instructions under `infrastructure/README.md`
- To create a new standard or autopilot cluster, follow the instructions in [`infrastructure/README.md`](https://github.com/GoogleCloudPlatform/ai-on-gke/blob/main/infrastructure/README.md)
- Alternatively, you can set the `create_cluster` variable to true in `workloads.tfvars` to provision a new GKE cluster. This will default to creating a GKE Autopilot cluster; if you want to provision a standard cluster you must also set `autopilot_cluster` to false.

3. This module is configured to use Identity Aware Proxy (IAP) as default authentication method for JupyterHub. It expects the brand & the OAuth consent configured in your org. You can check the details here: [OAuth consent screen](https://console.cloud.google.com/apis/credentials/consent)

Expand All @@ -30,7 +31,7 @@ This code can also perform auto brand creation. Please check the details [below]
* Terraform
* Gcloud CLI

Jupyterhub server can use either local storage or GCS to store notebooks and other artifcts.
JupyterHub server can use either local storage or GCS to store notebooks and other artifcts.
To use GCS, create a bucket with your username. For example, when authenticating with IAP as [email protected], ensure your bucket name is `gcsfuse-<username>`

## Installation
Expand All @@ -43,7 +44,7 @@ To use GCS, create a bucket with your username. For example, when authenticating
cd ai-on-gke/applications/jupyter
```

2. Edit `workloads.tfvars` with your GCP settings. The `namespace` that you specify will become a K8s namespace for your Jupyterhub services. For more information about what the variables do visit [here](https://github.com/GoogleCloudPlatform/ai-on-gke/blob/main/applications/jupyter/variable_definitions.md)
2. Edit `workloads.tfvars` with your GCP settings. The `namespace` that you specify will become a K8s namespace for your JupyterHub services. For more information about what the variables do visit [here](https://github.com/GoogleCloudPlatform/ai-on-gke/blob/main/applications/jupyter/variable_definitions.md)

**Important Note:**
If using this with the Ray module (`applications/ray/`), it is recommended to use the same k8s namespace
Expand All @@ -55,12 +56,12 @@ for both i.e. set this to the same namespace as `applications/ray/workloads.tfva
| cluster_name | GKE Cluster Name | Yes |
| cluster_location | GCP Region | Yes |
| cluster_membership_id | Fleet membership name for GKE cluster. <br /> Required when using private clusters with Anthos Connect Gateway | |
| namespace | The namespace that Jupyterhub and rest of the other resources will be installed in. | Yes |
| namespace | The namespace that JupyterHub and rest of the other resources will be installed in. | Yes |
| gcs_bucket | GCS bucket to be used for Jupyter storage | |
| create_service_account | Create service accounts used for Workload Identity mapping | Yes |
| gcp_and_k8s_service_account | GCP service account used for Workload Identity mapping and k8s sa attached with workload | Yes |

For variables under `Jupyterhub with IAP`, please see the section below
For variables under `JupyterHub with IAP`, please see the section below

### Secure endpoint with IAP

Expand All @@ -78,7 +79,7 @@ See the example `.tfvars` files under `/applications/jupyter` for different bran

| Variable | Description | Default Value | Required |
| ------------------------ |--------------------------- |:-------------:|:--------:|
| add_auth | Enable IAP on Jupyterhub | true | Yes |
| add_auth | Enable IAP on JupyterHub | true | Yes |
| brand | Name of the brand used for creating IAP OAuth clients. Only one is allowed per project. View existing brands: `gcloud iap oauth-brands list`. Leave it empty to create a new brand. Uses [support_email](#support_email) | | |
| support_email | Support email assocated with the [brand](#brand). Used as a point of contact for consent for the ["OAuth Consent" in Cloud Console](https://console.cloud.google.com/apis/credentials/consent). Optional field if `brand` is empty. | | |
| default_backend_service | default_backend_service | | |
Expand All @@ -87,7 +88,7 @@ See the example `.tfvars` files under `/applications/jupyter` for different bran
| url_domain_name | This variable will only be used if [url_domain_addr](#url_domain_addr) is provided. It is the name associated with the domain provided by the user. Since we are using Ingress, it will require the `kubernetes.io/ingress.global-static-ip-name` annotation along with the name associated. | | |
| client_id | Client ID of an [OAuth 2.0 Client ID](https://console.cloud.google.com/apis/credentials) created by the user for enabling IAP. You must also input the [client_secret](#client_secret). If this variable is unset, the template will create an OAuth client for you - in this case, you must ensure the associated [brand](https://console.cloud.google.com/apis/credentials/consent) is `Internal` i.e. only principals within the organization can access the application. | | |
| client_secret | Client Secret associated with the [client_id](#client_id). This variable will only be used when the client id is filled out. | | |
| members_allowlist | Comma seperated values for users to be allowed access through IAP. Example values: `allAuthenticatedUsers` or `allAuthenticatedUsers,user:[email protected]` | allAuthenticatedUsers | |
| members_allowlist | Comma seperated values for users to be allowed access through IAP. Example values: `user:[email protected]` | | |


### Install
Expand All @@ -109,19 +110,17 @@ gcloud auth application-default login
- Should have `jupyter-proxy-public` in the name eg.: `k8s1-63da503a-jupyter-proxy-public-80-74043627`.
* Run `terraform apply --var-file=./workloads.tfvars`

## Using Jupyterhub
## Using JupyterHub

### If Auth with IAP is disabled

1. Extract the randomly generated password for Jupyterhub login
1. Extract the randomly generated password for JupyterHub login

```
terraform output password
terraform output jupyterhub_password
```

2. Visit [Services](https://console.cloud.google.com/kubernetes/discovery) section on the GKE console & open the external IP for the `proxy-public` service in the browser.

> **_NOTE:_** If there isn't an external IP for `proxy-public`, is it most likely due to authentication being enabled.
2. Setup port forwarding for the frontend: `kubectl port-forward service/proxy-public -n <namespace> 8081:80 &`, and open `localhost:8081` in a browser.

### If Auth with IAP is enabled

Expand All @@ -139,17 +138,17 @@ Please note there may be some propagation delay after adding IAP principals (5-1
### Setup Access

In order for users to login to Jupyterhub via IAP, their access needs to be configured. To allow access for users/groups:
In order for users to login to JupyterHub via IAP, their access needs to be configured. To allow access for users/groups:

1. Navigate to the [GCP IAP Cloud Console](https://console.cloud.google.com/security/iap) and select your backend-service for `<namespace>/proxy-public`.

2. Click on `Add Principal`, insert the username / group name and select under `Cloud IAP` with role `IAP-secured Web App User`. Once presmission is granted, these users / groups can login to Jupyterhub with IAP. Please note there may be some propagation delay after adding IAP principals (5-10 mins).
2. Click on `Add Principal`, insert the username / group name and select under `Cloud IAP` with role `IAP-secured Web App User`. Once presmission is granted, these users / groups can login to JupyterHub with IAP. Please note there may be some propagation delay after adding IAP principals (5-10 mins).

## Persistent Storage

Jupyterhub is configured to provide 2 choices for storage:
JupyterHub is configured to provide 2 choices for storage:

1. Default Jupyterhub Storage - `pd.csi.storage.gke.io` with reclaim policy `Delete`
1. Default JupyterHub Storage - `pd.csi.storage.gke.io` with reclaim policy `Delete`

2. GCSFuse - `gcsfuse.csi.storage.gke.io` uses GCS Buckets and require users to pre-create buckets with name format `gcsfuse-{username}`

Expand Down Expand Up @@ -180,6 +179,7 @@ This example is adapted from Ray AIR's examples [here](https://docs.ray.io/en/la
## Auto Brand creation and IAP enablement

**IMPORTANT** If you enable automatic brand creation, only `Internal` brand will be created, allowing only the users under the same org as the project to access the application.
Make sure [Policy for Restrict Load Balancer Creation Based on Load Balancer Types](https://cloud.google.com/load-balancing/docs/org-policy-constraints) allows EXTERNAL_HTTP_HTTPS.

Ensure that the following variables within `workloads.tfvars` are set:

Expand All @@ -194,4 +194,4 @@ This module uses `<ip>.nip.io` as the domain name with a global static ipv4 addr

## Additional Information

For more information about Jupyterhub profiles and the preset profiles visit [here](https://github.com/GoogleCloudPlatform/ai-on-gke/blob/main/applications/jupyter/profiles.md)
For more information about JupyterHub profiles and the preset profiles visit [here](https://github.com/GoogleCloudPlatform/ai-on-gke/blob/main/applications/jupyter/profiles.md)
82 changes: 53 additions & 29 deletions applications/jupyter/main.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,42 @@ data "google_project" "project" {
project_id = var.project_id
}

## Enable Required GCP Project Services APIs
module "project-services" {
source = "terraform-google-modules/project-factory/google//modules/project_services"
version = "~> 14.5"

project_id = var.project_id
disable_services_on_destroy = false
disable_dependent_services = false
activate_apis = flatten([
"autoscaling.googleapis.com",
"cloudbuild.googleapis.com",
"cloudresourcemanager.googleapis.com",
"compute.googleapis.com",
"config.googleapis.com",
"connectgateway.googleapis.com",
"container.googleapis.com",
"containerfilesystem.googleapis.com",
"dns.googleapis.com",
"gkehub.googleapis.com",
"iamcredentials.googleapis.com",
"logging.googleapis.com",
"monitoring.googleapis.com",
"pubsub.googleapis.com",
"servicenetworking.googleapis.com",
"serviceusage.googleapis.com",
"sourcerepo.googleapis.com",
"iap.googleapis.com"
])
}

module "infra" {
source = "../../infrastructure"
count = var.create_cluster ? 1 : 0

project_id = var.project_id
cluster_name = var.cluster_name
cluster_name = local.cluster_name
cluster_location = var.cluster_location
autopilot_cluster = var.autopilot_cluster
private_cluster = var.private_cluster
Expand All @@ -39,25 +69,27 @@ module "infra" {
subnetwork_name = "default"
cpu_pools = var.cpu_pools
enable_gpu = false
depends_on = [module.project-services]
}

data "google_container_cluster" "default" {
count = var.create_cluster ? 0 : 1
name = var.cluster_name
location = var.cluster_location
}

locals {
endpoint = var.create_cluster ? "https://${module.infra[0].endpoint}" : "https://${data.google_container_cluster.default[0].endpoint}"
ca_certificate = var.create_cluster ? base64decode(module.infra[0].ca_certificate) : base64decode(data.google_container_cluster.default[0].master_auth[0].cluster_ca_certificate)
private_cluster = var.create_cluster ? var.private_cluster : data.google_container_cluster.default[0].private_cluster_config.0.enable_private_endpoint
cluster_membership_id = var.cluster_membership_id == "" ? var.cluster_name : var.cluster_membership_id
enable_autopilot = var.create_cluster ? var.autopilot_cluster : data.google_container_cluster.default[0].enable_autopilot
host = local.private_cluster ? "https://connectgateway.googleapis.com/v1/projects/${data.google_project.project.number}/locations/${var.cluster_location}/gkeMemberships/${local.cluster_membership_id}" : local.endpoint
count = var.create_cluster ? 0 : 1
name = var.cluster_name
location = var.cluster_location
depends_on = [module.project-services]
}

locals {
endpoint = var.create_cluster ? "https://${module.infra[0].endpoint}" : "https://${data.google_container_cluster.default[0].endpoint}"
ca_certificate = var.create_cluster ? base64decode(module.infra[0].ca_certificate) : base64decode(data.google_container_cluster.default[0].master_auth[0].cluster_ca_certificate)
private_cluster = var.create_cluster ? var.private_cluster : data.google_container_cluster.default[0].private_cluster_config.0.enable_private_endpoint
cluster_membership_id = var.cluster_membership_id == "" ? local.cluster_name : var.cluster_membership_id
enable_autopilot = var.create_cluster ? var.autopilot_cluster : data.google_container_cluster.default[0].enable_autopilot
host = local.private_cluster ? "https://connectgateway.googleapis.com/v1/projects/${data.google_project.project.number}/locations/${var.cluster_location}/gkeMemberships/${local.cluster_membership_id}" : local.endpoint
kubernetes_namespace = var.goog_cm_deployment_name != "" ? "${var.goog_cm_deployment_name}-${var.kubernetes_namespace}" : var.kubernetes_namespace
cluster_name = var.goog_cm_deployment_name != "" ? "${var.goog_cm_deployment_name}-${var.cluster_name}" : var.cluster_name
workload_identity_service_account = var.goog_cm_deployment_name != "" ? "${var.goog_cm_deployment_name}-${var.workload_identity_service_account}" : var.workload_identity_service_account
jupyterhub_default_uri = "https://console.cloud.google.com/kubernetes/service/${var.cluster_location}/${local.cluster_name}/${local.kubernetes_namespace}/proxy-public/overview?project=${var.project_id}"
}

provider "kubernetes" {
Expand Down Expand Up @@ -101,33 +133,26 @@ module "gcs" {
module "namespace" {
source = "../../modules/kubernetes-namespace"
providers = { helm = helm.jupyter }
namespace = var.kubernetes_namespace
namespace = local.kubernetes_namespace
create_namespace = true
}

# IAP Section: Enabled the IAP service
resource "google_project_service" "project_service" {
count = var.add_auth ? 1 : 0
project = var.project_id
service = "iap.googleapis.com"

disable_dependent_services = false
disable_on_destroy = false
}

# Creates jupyterhub
module "jupyterhub" {
source = "../../modules/jupyter"
providers = { helm = helm.jupyter, kubernetes = kubernetes.jupyter }
project_id = var.project_id
namespace = var.kubernetes_namespace
namespace = local.kubernetes_namespace
additional_labels = var.additional_labels
workload_identity_service_account = local.workload_identity_service_account
gcs_bucket = var.gcs_bucket
autopilot_cluster = local.enable_autopilot
notebook_image = "jupyter/tensorflow-notebook"
notebook_image_tag = "python-3.10"

# IAP Auth parameters
add_auth = var.add_auth
brand = var.brand
create_brand = var.create_brand
support_email = var.support_email
client_id = var.client_id
client_secret = var.client_secret
Expand All @@ -137,8 +162,7 @@ module "jupyterhub" {
k8s_backend_config_name = var.k8s_backend_config_name
k8s_backend_service_name = var.k8s_backend_service_name
k8s_backend_service_port = var.k8s_backend_service_port
url_domain_addr = var.url_domain_addr
url_domain_name = var.url_domain_name
members_allowlist = var.members_allowlist
domain = var.domain
members_allowlist = var.members_allowlist != "" ? split(",", var.members_allowlist) : []
depends_on = [module.gcs, module.namespace]
}
Loading

0 comments on commit b6ddc5a

Please sign in to comment.