Add Self-Hosted (Customer Managed) Data Plane Installation guide (#717)

Co-authored-by: Rob Dominguez <[email protected]>
Co-authored-by: Shahidh K Muhammed <[email protected]>

docs/deployment/private/data-plane-collaboration.mdx

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 4
+sidebar_position: 5
 sidebar_label: Data Plane Collaboration
 description:
   "Learn how to use the data plane collaboration feature for Private DDN."

docs/deployment/private/index.mdx

@@ -21,3 +21,4 @@ other connectors. Hasura communicates with your sources over a dedicated private
 - [Hasura-Hosted (VPC)](/deployment/private/hasura-hosted.mdx) private deployments
 - [Self-Hosted (BYOC)](/deployment/private/self-hosted.mdx) private deployments
 - [Data Plane Collaboration](/deployment/private/data-plane-collaboration.mdx)
+- [Self-Hosted (Customer Managed) Data Plane Installation](/deployment/private/self-hosted-deployment.mdx)

docs/deployment/private/self-hosted-deployment.mdx

@@ -0,0 +1,358 @@
+---
+sidebar_position: 4
+sidebar_label: Self-Hosted (Customer Managed) Data Plane Installation Guide
+description: "Learn how to install a Self-Hosted (Customer Managed) Data Plane."
+keywords:
+  - hasura ddn
+  - enterprise ddn
+  - private ddn
+---
+
+import Thumbnail from "@site/src/components/Thumbnail";
+
+# Self hosted (Customer Managed) Data Plane Installation Guide
+
+:::info
+
+Documentation here targets customers who want to self host and self manage their clusters as well as their workloads.  Here, you will find a full set of instructions, which takes you from local development all the way to having your workloads running under your Kubernetes hosted data plane.
+:::
+
+## Prerequisites
+
+Before continuing, ensure you go through the following checklist and confirm that you meet all the requirements
+
+- [Docker v2.27.1](/getting-started/build/prerequisites/#step-3-install-docker-compose-v2271-or-greater) (Or greater)
+- [DDN CLI](/getting-started/build/prerequisites#step-1-install-and-authorize-the-hasura-cli) (Latest)
+- [Helm3](https://helm.sh/docs/intro/install/) (Prefer latest)
+- [Hasura VS Code Extension](/getting-started/build/prerequisites#install-lsp) (Recommended, but not required)
+- Access to a Kubernetes cluster
+- Ability to build and push images that can then be pulled down from the Kubernetes cluster
+- A user account on the Hasura DDN Control Plane
+- A Data Plane id & key created by the Hasura team.  These will be referenced as `<data-plane-id>` and `<data-plane-key>`
+
+## Step 1. Local development
+
+:::note
+
+In this step, you will be setting up your local development.  You will create a supergraph, add connector(s) to it and perform a local build.
+
+For more details regarding local development, you may reference our [Getting Started Guide](/getting-started/overview/).
+:::
+
+```bash title="Login with the DDN CLI."
+ddn auth login
+```
+
+```bash title="Create a local directory where your supergraph will be created.  Afterwards, cd into it."
+mkdir hasura_project && cd hasura_project
+```
+
+```bash title="Initialize supergraph."
+ddn supergraph init .
+```
+
+```bash title="Add a connector.  Here, we are using the -i flag for interactive mode."
+ddn connector init -i
+```
+
+```bash title="Introspect the data source.  Substitute <connector-name> with name you chose for your connector above."
+ddn connector introspect <connector-name>
+```
+
+```bash title="Add models, commands and relationships based on the output from previous command.  Substitute <connector-name> with name you chose for your connector above."
+ddn model add <connector-name> "*"
+ddn command add <connector-name> "*"
+ddn relationship add <connector-name> "*"
+```
+
+```bash title="Build the supergraph locally."
+ddn supergraph build local
+```
+
+```bash title="Run docker.  Ensure all containers start up successfully."
+ddn run docker-start
+```
+
+```bash title="Verify via local console."
+ddn console --local
+```
+
+At this point, you have connected and introspected a data source and built your supergraph locally.  Verify that everything is working as expected before moving on to the next section.
+
+## Step 2. Build connector(s)
+
+:::warning Building images with proper OS/Arch
+
+Ensure that you are building the image with the correct OS/arch which would enable the image to run properly under your Kubernetes cluster
+:::
+
+:::note
+
+Repeat the steps below for each connector within your supergraph.
+:::
+
+```bash title="Build image via docker compose.  Find the <service-name> from app/connector/<connector>/compose.yaml"
+docker compose build <service-name>
+```
+
+```bash title="Re-tag the image"
+docker tag <root-folder-name>-app_<connector> <your_registry>/<connector>:<your_tag>
+```
+
+```bash title="Push the image to your registry."
+docker push <your_registry>/<connector>:<your_tag>
+```
+
+## Step 3. Deploy connector(s) with Helm
+
+:::info Hasura DDN Helm Repo
+
+Our DDN Helm Repo can be found [here](https://github.com/hasura/ddn-helm-charts/tree/main).  Ensure that you run through the `Get Started` section there first before you attempt to install any Helm charts.
+
+Contact the Hasura team if you don't see a Helm chart available for your connector.
+:::
+
+Execute `helm search repo hasura-ddn` in order to find the appropriate Helm chart for your connector.
+
+A typical connector Helm install command would look like this:
+
+:::info
+
+- `<connector-helm-release-name>`: Helm release name for your connector.
+
+- `<connector-type>`: The connector type.  Select the appropriate value from [here](https://raw.githubusercontent.com/hasura/ddn-helm-charts/a6b8f3506c02ea298fd3d4b20fba701f31a9998b/CONNECTORS).
+
+- `<data-plane-id>`: Data Plane id, provided by the Hasura team.
+
+- `<data-plane-key>`: Data Plane key, provided by the Hasura team.
+
+- `<connector_env_variable>`: Connector Env variable name and corresponding `<value>`.  Repeat setting this for all connector environment variables which you need to pass along.
+:::
+
+```bash title="Connector Helm install"
+helm upgrade --install <connector-helm-release-name> \
+  --set image.repository="my_repo/ndc-mongodb" \
+  --set image.tag="my_custom_image_tag" \
+  --set dataPlane.id="<data-plane-id>" \
+  --set dataPlane.key="<data-plane-key>" \
+  --set connectorEnvVars.<connector_env_variable>="<value>" \
+  hasura-ddn/<connector-type>
+```
+
+## Step 4. Create a cloud project
+
+Ensure that you are running the commands here from the root of your supergraph.
+
+```bash title="Create cloud project."
+ddn project init --data-plane-id <data-plane-id>
+```
+
+This command will create a cloud project and will report back as to what the name of your cloud project is.  We will reference this name going forward as `<ddn-project-name>`.
+
+
+## Step 5. Update .env.cloud with connector URLs
+
+Next, you will need to access the `.env.cloud` file which was generated at the root of your supergraph.
+
+For **each** READ_URL & WRITE_URL, you will need to make the necessary adjustments to ensure that your v3-engine (Which will be running in your Kubernetes cluster) is properly configured to reach each one of your connectors.  The current values that these environment variables point to are for local connectivity.
+
+The URLs of these environment variables needs to be structured as shown below.
+
+:::info
+
+
+URL: **http://`<connector-helm-release-name>`-`<connector-type>`.`<namespace>`:8080**
+
+- `<connector-helm-release-name>`: Helm release name for your connector.
+
+- `<connector-type>`: The connector type.  Select the appropriate value from [here](https://raw.githubusercontent.com/hasura/ddn-helm-charts/a6b8f3506c02ea298fd3d4b20fba701f31a9998b/CONNECTORS).
+
+- `<namespace>`: Namespace where your connector was deployed to.
+:::
+
+## Step 6. Create a cloud build
+
+After making the necessary changes to your `.env.cloud` file, run the below command.  This will create a cloud build and will also generate the necessary artifacts which will later on be consumed by your v3-engine. 
+
+```bash title="Create a build for your cloud project."
+ddn supergraph build create --self-hosted-data-plane --output-dir build-data --project <ddn-project-name> --out json
+```
+
+At this point, take note of the `<build_version>` and `<observability_hostname>` which will be outputted here.  You will need these later.
+
+## Step 7. Build v3-engine
+
+:::warning Building images with proper OS/Arch
+
+Ensure that you are building the image with the correct OS/arch which would enable the image to run properly under your Kubernetes cluster
+:::
+
+Ensure that you are running the commands from the root of your supergraph.
+
+```bash title="Create a Dockerfile for v3-engine."
+cat <<EOF >> Dockerfile
+FROM ghcr.io/hasura/v3-engine
+COPY ./build-data /md/
+EOF
+```
+
+```bash title="Build the image via docker build.  Tag this image with your own registry and a custom tag of your choosing."
+docker build -t <your_registry>/v3-engine:<your_tag> .
+```
+
+```bash title="Push the image to your registry."
+docker push <your_registry>/v3-engine:<your_tag>
+```
+
+## Step 8. Deploy v3-engine with Helm
+
+See the DDN Helm Repo [v3-engine](https://github.com/hasura/ddn-helm-charts/tree/main/charts/v3-engine) section for full documentation.  A typical v3-engine Helm installation would look like this:
+
+:::info
+
+- `<v3-engine-helm-release-name>`: Helm release name for v3-engine.
+
+- `<observability_hostname>`: Observability hostname.  This was returned in output when `ddn supergraph build create` was executed.
+
+- `<data-plane-id>`: Data Plane id, provided by the Hasura team.
+
+- `<data-plane-key>`: Data Plane key, provided by the Hasura team.
+:::
+
+```bash title="v3-engine helm install"
+helm upgrade --install <v3-engine-helm-release-name> \
+  --set image.repository="my_repo/v3-engine" \
+  --set image.tag="my_custom_image_tag" \
+  --set observability.hostName="<observability_hostname>" \
+  --set dataPlane.id="<data-plane-id>" \
+  --set dataPlane.key="<data-plane-key>" \
+  hasura-ddn/v3-engine
+```
+
+## Step 9. Create build specific ingress
+
+:::note
+
+Everytime you create a new build via `ddn supergraph build create` command, execute the steps below.
+:::
+
+If you're using <b>nginx-ingress</b> and <b>cert-manager</b>, you can deploy using the below manifest.  Ensure that you modify this accordingly
+
+:::info
+
+
+- `<build_version>`: This was part of the output when you ran `ddn supergraph build create` command.
+
+- `<domain>`: Domain which will be used for accessing this ingress.  This could be constructed in the following format: **https://`<build_version>`.`<your_fqdn>`**.
+
+- `<namespace>`: Namespace where your v3-engine was deployed to.
+
+- `<v3-engine-helm-release-name>`: Helm release name for your v3-engine.
+:::
+
+```bash
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
+  labels:
+    app: v3-engine-<build_version>
+  name: v3-engine-<build_version>
+  namespace: <namespace>
+spec:
+  ingressClassName: nginx
+  rules:
+  - host: <domain>
+    http:
+      paths:
+      - backend:
+          service:
+            name: <v3-engine-helm-release-name>-v3-engine
+            port:
+              number: 3000
+        path: /
+        pathType: ImplementationSpecific
+  tls:
+  - hosts:
+    - <domain>
+    secretName: <domain>-tls-certs
+```
+
+Next, you will be running the command below in order to record the ingress URL within Hasura's Control Plane.
+
+:::info
+
+- `<ingress-url>`: Domain name, chosen above and prepended with protocol.
+:::
+
+```bash title="Record the ingress URL for your build version."
+ddn supergraph build set-self-hosted-engine-url <ingress-url> --build-version <build_version> --project <ddn-project-name>
+```
+
+## Step 10. Apply and promote a build to project's API
+
+:::note
+
+Everytime you want to apply and promote a build to the project's API, execute the steps below.
+:::
+
+**NOTE:** We are once again using an example of an ingress object which will work provided you have <b>nginx</b> and <b>cert-manager</b> installed on your cluster.
+
+:::info
+
+
+- `<namespace>`: Namespace where your v3-engine was deployed to.
+
+- `<domain>`: Domain which will be used for accessing this ingress.  This could be constructed in the following format: **https://`<prod-api>`.`<your_fqdn>`**.
+
+- `<v3-engine-helm-release-name>`: Helm release name for your v3-engine.  Choose an appropriate v3-engine Helm release name that you want to be applied/promoted to the project's API.
+:::
+
+```bash
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
+  labels:
+    app: v3-engine
+  name: v3-engine
+  namespace: <namespace>
+spec:
+  ingressClassName: nginx
+  rules:
+  - host: <domain>
+    http:
+      paths:
+      - backend:
+          service:
+            name: <v3-engine-helm-release-name>-v3-engine
+            port:
+              number: 3000
+        path: /
+        pathType: ImplementationSpecific
+  tls:
+  - hosts:
+    - <domain>
+    secretName: <domain>-tls-certs
+```
+
+Next, you will be running the command below in order to record the project's API URL within Hasura's Control Plane.
+
+:::tip
+
+**You will only need to do this once**.
+
+- `<ingress-url>`: Domain name, chosen above and prepended with protocol.
+:::
+
+```bash title="Record the ingress URL for your project's API."
+ddn project set-self-hosted-engine-url <ingress-url>
+```
+
+## Step 11. View API via console
+
+Access [Hasura console](https://console.hasura.io) and locate your cloud project.  Access your project and verify your deployment.

docs/deployment/private/self-hosted.mdx

@@ -32,6 +32,8 @@ There are two models of deployment with Self-Hosted DDN.
    you control the uptime, upgrades, etc. All API traffic stays within your network and does not leave the boundaries of
    your system.
 
+   Installation docs related to deployment of a self hosted, customer managed data plane are located [here](/deployment/private/self-hosted-deployment.mdx)
+
 In both cases, the [control plane](/deployment/architecture.mdx#control-plane) is always hosted by Hasura.
 
 <Thumbnail src="/img/deployment/private_ddn_customer_hosted.png" alt="Private DDN Architecture Customer Hosted" />