The detailed explanation of example code is here
- This will execute 6 steps. - - 1. init - - - Import packages -
example code
- - ```go - package main - - import ( - "context" - "encoding/json" - "flag" - "time" - - "github.com/kpango/fuid" - "github.com/kpango/glg" - "github.com/vdaas/vald-client-go/v1/payload" - "github.com/vdaas/vald-client-go/v1/vald" - - "gonum.org/v1/hdf5" - "google.golang.org/grpc" - ) - ``` - -
example code
- - ```go - const ( - insertCount = 400 - testCount = 20 - ) - ``` - -
example code
- - ```go - const ( - datasetPath string - grpcServerAddr string - indexingWaitSeconds uint - ) - ``` - -
example code
- - ```go - func init() { - flag.StringVar(&datasetPath, "path", "fashion-mnist-784-euclidean.hdf5", "set dataset path") - flag.StringVar(&grpcServerAddr, "addr", "127.0.0.1:8081", "set gRPC server address") - flag.UintVar(&indexingWaitSeconds, "wait", 60, "set indexing wait seconds") - flag.Parse() - } - ``` - -
example code
- - ```go - ids, train, test, err := load(datasetPath) - if err != nil { - glg.Fatal(err) - } - ``` - -
example code
- - ```go - ctx := context.Background() - - conn, err := grpc.NewClient(grpcServerAddr, grpc.WithInsecure()) - if err != nil { - glg.Fatal(err) - } - - client := vald.NewValdClient(conn) - ``` - -
example code
- - ```go - for i := range ids [:insertCount] { - _, err := client.Insert(ctx, &payload.Insert_Request{ - Vector: &payload.Object_Vector{ - Id: ids[i], - Vector: train[i], - }, - Config: &payload.Insert_Config{ - SkipStrictExistCheck: true, - }, - }) - if err != nil { - glg.Fatal(err) - } - if i%10 == 0 { - glg.Infof("Inserted %d", i) - } - } - ``` - -
example code
- - ```go - wt := time.Duration(indexingWaitSeconds) * time.Second - glg.Infof("Wait %s for indexing to finish", wt) - time.Sleep(wt) - ``` - -
example code
- - ```go - glg.Infof("Start search %d times", testCount) - for i, vec := range test[:testCount] { - res, err := client.Search(ctx, &payload.Search_Request){ - Vector: vec, - Config: &payload.Search_Config{ - Num: 10, - Radius: -1, - Epsilon: 0.1, - Timeout: 100000000, - } - } - if err != nil { - glg.Fatal(err) - } - - b, _ := json.MarshalIndent(res.GetResults(), "", " ") - glg.Infof("%d - Results : %s\n\n", i+1, string(b)) - time.Sleep(1 * time.Second) - } - ``` - -
example code
- - ```go - for i := range ids [:insertCount] { - _, err := client.Remove(ctx, &payload.Remove_Request{ - Id: &payload.Object_ID{ - Id: ids[i], - }, - }) - if err != nil { - glg.Fatal(err) - } - if i%10 == 0 { - glg.Infof("Removed %d", i) - } - } - ``` - -
Installation command for Helm
```bash -curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash +curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash ```
```go ctx := context.Background() - conn, err := grpc.NewClient(grpcServerAddr, grpc.WithTransportCredentials(insecure.NewCredentials())) + conn, err := grpc.DialContext(ctx, grpcServerAddr, grpc.WithTransportCredentials(insecure.NewCredentials())) if err != nil { glg.Fatal(err) } @@ -425,11 +425,11 @@ If you are interested, please refer to [SDKs](/docs/v1.7/user-guides/sdks).
1. Remove - - Remove 200 indexed training datasets from the Vald agent. + - Remove 400 indexed training datasets from the Vald agent.
example code
```go - for i := range ids [:removeCount] { + for i := range ids [:insertCount] { _, err := client.Remove(ctx, &payload.Remove_Request{ Id: &payload.Object_ID{ Id: ids[i], @@ -446,20 +446,6 @@ If you are interested, please refer to [SDKs](/docs/v1.7/user-guides/sdks).
example code
- - ```go - _, err := client.Flush(ctx, &payload.Flush_Request{}) - if err != nil { - glg.Fatal(err) - } - ``` - -
Installation command for Helm
```bash -curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash +curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash ```
-
-## Requirements
-
-- Vald: v1.8 ~
-- Kubernetes: v1.27 ~
-- Go: v1.20 ~
-- Helm: v3 ~
-- libhdf5 (_only required to get started_)
-
-Helm is used to deploy Vald cluster on your Kubernetes and HDF5 is used to decode the sample data file to run the example code.
-
-If Helm or HDF5 is not installed, please install [Helm](https://helm.sh/docs/intro/install) and [HDF5](https://www.hdfgroup.org/).
-
-Installation command for Helm
- -```bash -curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash -``` - -
Installation command for HDF5
- -```bash -# yum -yum install -y hdf5-devel - -# apt -apt-get install libhdf5-serial-dev - -# homebrew -brew install hdf5 -``` - -
vald-01 Namespace
- - ```bash - kubectl get pods -n vald-01 - NAME READY STATUS RESTARTS AGE - vald-agent-ngt-0 1/1 Running 0 2m41s - vald-agent-ngt-2 1/1 Running 0 2m41s - vald-agent-ngt-3 1/1 Running 0 2m41s - vald-agent-ngt-4 1/1 Running 0 2m41s - vald-agent-ngt-5 1/1 Running 0 2m41s - vald-agent-ngt-1 1/1 Running 0 2m41s - vald-discoverer-77967c9697-brbsp 1/1 Running 0 2m41s - vald-lb-gateway-587879d598-xmws7 1/1 Running 0 2m41s - vald-lb-gateway-587879d598-dzn9c 1/1 Running 0 2m41s - vald-manager-index-56d474c848-wkh6b 1/1 Running 0 2m41s - vald-lb-gateway-587879d598-9wb5q 1/1 Running 0 2m41s - vald-mirror-gateway-6df75cf7cf-gzcr4 1/1 Running 0 2m26s - vald-mirror-gateway-6df75cf7cf-vjbqx 1/1 Running 0 2m26s - vald-mirror-gateway-6df75cf7cf-c2g7t 1/1 Running 0 2m41s - ``` - -
vald-02 Namespace
- - ```bash - kubectl get pods -n vald-02 - NAME READY STATUS RESTARTS AGE - vald-agent-ngt-0 1/1 Running 0 2m52s - vald-agent-ngt-1 1/1 Running 0 2m52s - vald-agent-ngt-2 1/1 Running 0 2m52s - vald-agent-ngt-4 1/1 Running 0 2m52s - vald-agent-ngt-5 1/1 Running 0 2m52s - vald-agent-ngt-3 1/1 Running 0 2m52s - vald-discoverer-8cfcff76-vlmpg 1/1 Running 0 2m52s - vald-lb-gateway-54896f9f49-wtlcv 1/1 Running 0 2m52s - vald-lb-gateway-54896f9f49-hbklj 1/1 Running 0 2m52s - vald-manager-index-676855f8d7-bb4wb 1/1 Running 0 2m52s - vald-lb-gateway-54896f9f49-kgrdf 1/1 Running 0 2m52s - vald-mirror-gateway-6598cf957-t2nz4 1/1 Running 0 2m37s - vald-mirror-gateway-6598cf957-wr448 1/1 Running 0 2m52s - vald-mirror-gateway-6598cf957-jdd6q 1/1 Running 0 2m37s - ``` - -
vald-03 Namespace
- - ```bash - kubectl get pods -n vald-03 - NAME READY STATUS RESTARTS AGE - vald-agent-ngt-0 1/1 Running 0 2m46s - vald-agent-ngt-1 1/1 Running 0 2m46s - vald-agent-ngt-2 1/1 Running 0 2m46s - vald-agent-ngt-3 1/1 Running 0 2m46s - vald-agent-ngt-4 1/1 Running 0 2m46s - vald-agent-ngt-5 1/1 Running 0 2m46s - vald-discoverer-879867b44-8m59h 1/1 Running 0 2m46s - vald-lb-gateway-6c8c6b468d-ghlpx 1/1 Running 0 2m46s - vald-lb-gateway-6c8c6b468d-rt688 1/1 Running 0 2m46s - vald-lb-gateway-6c8c6b468d-jq7pl 1/1 Running 0 2m46s - vald-manager-index-5596f89644-xfv4t 1/1 Running 0 2m46s - vald-mirror-gateway-7b95956f8b-l57jz 1/1 Running 0 2m31s - vald-mirror-gateway-7b95956f8b-xd9n5 1/1 Running 0 2m46s - vald-mirror-gateway-7b95956f8b-dnxbb 1/1 Running 0 2m31s - ``` - -
Example output
- -```bash -kubectl get vmt -A -o wide -NAMESPACE NAME HOST PORT STATUS LAST TRANSITION TIME AGE -vald-03 mirror-target-01 vald-mirror-gateway.vald-01.svc.cluster.local 8081 Connected 2023-05-22T02:07:51Z 19m -vald-03 mirror-target-02 vald-mirror-gateway.vald-02.svc.cluster.local 8081 Connected 2023-05-22T02:07:51Z 19m -vald-02 mirror-target-3296010438411762394 vald-mirror-gateway.vald-01.svc.cluster.local 8081 Connected 2023-05-22T02:07:53Z 19m -vald-02 mirror-target-12697587923462644654 vald-mirror-gateway.vald-03.svc.cluster.local 8081 Connected 2023-05-22T02:07:53Z 19m -vald-01 mirror-target-13698925675968803691 vald-mirror-gateway.vald-02.svc.cluster.local 8081 Connected 2023-05-22T02:07:53Z 19m -vald-01 mirror-target-17825710563723389324 vald-mirror-gateway.vald-03.svc.cluster.local 8081 Connected 2023-05-22T02:07:53Z 19m -``` - -
-When introducing production, we recommend that you actually measure how many resources are required for verification.
-
-
## Vald cluster view
The essential point at the Vald cluster view is the hardware specification, especially RAM.
The Vald cluster, especially Vald Agent components, requires much RAM capacity because the vector index is stored in memory.
-The minimum required memory for each vector (bit) is:
-
-```bash
-// minimum required bits of vector
-{ oid (64bit) + timestamp (64bit) + uuid (user defined) } * 2 + { dimension * 64 } + { the creation edge size + the search edge size } * 8
-```
-
-Considering the `index size` and `index_replica`, it is easy to figure out the minimum required RAM capacity by the following formula.
+It is easy to figure out the minimum required RAM capacity by the following formula.
```bash
-{ minimum required bits of vector } * { the index size } * { index_replica }
+( { the dimension vector } × { bit number of vector } + { the bit of vectors ID string } ) × { the maximum number of the vector } × { the index replica }
```
-For example, you want to insert 1 million vectors with 900 dimensions with 32 byte (256 bit) UUID, the index replica is 3, `creation edge size` is 20, and `search edge size` is 10, the minimum required RAM capacity is:
+For example, if you want to insert 1 million vectors with 900 dimensions and the object type is 32-bit with 32 byte (256 bit) ID, and the index replica is 3, the minimum required RAM capacity is:
```bash
-{(64 + 64 + 256) × 2 + (900 × 64) + (20 + 10) × 8 } × 1,000,000 × 3 = 175,824,000,000 (bit) = 21.978 (GB)
+(900 × 32 + 256 ) × 1,000,000 × 3 = 8,7168,000,000 (bit) = 10.896 (GB)
```
It is just the minimum required RAM for indexing.
@@ -50,12 +39,12 @@ Considering the margin of RAM capacity, the minimum RAM capacity should be less
Therefore, the actual minimum RAM capacity will be:
```bash
-175,824,000,000 (bit) / 0.6 = 293,040,000,000 (bit) = 36.63 (GB)
+8,7168,000,000 (bit) / 0.6 = 145,280,000,000 (bit) = 18.16 (GB)
```
In the production usage, memory usage may be not enough in the minimum required RAM.
-Because for example, there are a noisy problem, high memory usage for createIndex (indexing on memory), high traffic needs more memory, etc. +E.g., there are a noisy problem, high memory usage for createIndex (indexing on memory), high traffic needs more memory, etc.
## Kubernetes cluster view
diff --git a/content/docs/v1.7/user-guides/client-api-config.md b/content/docs/v1.7/user-guides/client-api-config.md
index 232612d8c..12c910968 100644
--- a/content/docs/v1.7/user-guides/client-api-config.md
+++ b/content/docs/v1.7/user-guides/client-api-config.md
@@ -1,6 +1,6 @@
---
title: "Client Api Config_v1.7/User Guides"
-date: 2024-10-16T16:05:59+09:00
+date: 2025-04-17T05:06:32Z
draft: false
weight: 1000
description: Shows more detailed explanation about APIs configuration
@@ -57,7 +57,7 @@ func main() {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
target := "localhost:8080"
- conn, err := grpc.NewClient(target)
+ conn, err := grpc.DialContext(ctx, target)
if err != nil {
panic(err)
}
@@ -173,7 +173,7 @@ func example() {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
target := "localhost:8080"
- conn, err := grpc.NewClient(target)
+ conn, err := grpc.DialContext(ctx, target)
if err != nil {
panic(err)
}
@@ -297,7 +297,7 @@ func example() {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
target := "localhost:8080"
- conn, err := grpc.NewClient(target)
+ conn, err := grpc.DialContext(ctx, target)
if err != nil {
panic(err)
}
@@ -481,7 +481,7 @@ func main() {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
target := "localhost:8080"
- conn, err := grpc.NewClient(target)
+ conn, err := grpc.DialContext(ctx, target)
if err != nil {
panic(err)
}
@@ -664,7 +664,7 @@ func example() {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
target := "localhost:8080"
- conn, err := grpc.NewClient(target)
+ conn, err := grpc.DialContext(ctx, target)
if err != nil {
panic(err)
}
diff --git a/content/docs/v1.7/user-guides/cluster-role-binding.md b/content/docs/v1.7/user-guides/cluster-role-binding.md
index b41fe5a82..6d014563a 100644
--- a/content/docs/v1.7/user-guides/cluster-role-binding.md
+++ b/content/docs/v1.7/user-guides/cluster-role-binding.md
@@ -1,6 +1,6 @@
---
title: "Cluster Role Binding_v1.7/User Guides"
-date: 2024-10-16T16:05:59+09:00
+date: 2025-04-17T05:06:32Z
draft: false
weight: 500
description: Configures cluster role for Vald cluster
@@ -144,105 +144,6 @@ If you disable these configurations, the Vald Discoverer will not work, and the
If you want to modify or disable these configurations, you need to grant the [cluster role configuration](https://github.com/vdaas/vald/blob/main/k8s/discoverer/clusterrole.yaml) and bind it to the Vald Discoverer to retrieve required information to operate the Vald cluster.
-## Configuration for Vald Mirror Gateway
-
-The Vald Mirror Gateway requires configuration on cluster role and cluster role binding to create/update/retrive [ValdMirrorTarget](https://vald.vdaas.org/docs/user-guides/mirroring-configuration/) resource from the Kubernetes Cluster.
-
-In this section, we will describe how to configure it and how to customize these configurations.
-
-### Cluster role configuration for Vald Mirror Gateway
-
-By looking at the [cluster role configuration](https://github.com/vdaas/vald/blob/main/k8s/gateway/mirror/clusterrole.yaml), the access right of the following resources are granted to the cluster role `gateway-mirror`.
-
-```yaml
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
- name: gateway-mirror
----
-rules:
- - apiGroups:
- - vald.vdaas.org
- resources:
- - valdmirrortargets
- verbs:
- - create
- - update
- - delete
- - get
- - list
- - watch
- - patch
- - apiGroups:
- - vald.vdaas.org
- resources:
- - valdmirrortargets/status
- verbs:
- - create
- - update
- - get
- - list
- - patch
- - apiGroups:
- - vald.vdaas.org
- resources:
- - valdmirrortargets/finalizers
- verbs:
- - update
-```
-
-### Cluster role binding configuration for Vald Mirror Gateway
-
-The cluster role binding configuration binds the cluster role `gateway-mirror` described in the previous section to the service account `gateway-mirror` according to the [configuration file](https://github.com/vdaas/vald/blob/main/k8s/gateway/mirror/clusterrolebinding.yaml).
-
-```yaml
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
- name: gateway-mirror
- ...
-roleRef:
- apiGroup: rbac.authorization.k8s.io
- kind: ClusterRole
- name: gateway-mirror
-subjects:
- - kind: ServiceAccount
- name: gateway-mirror
- namespace: default
-```
-
-When the role binds to the service account, the access right of the role will be granted to the service account.
-In this case, all the access rights of the role `gateway-mirror` will be granted to the service account `gateway-mirror`.
-
-### Customize cluster role and cluster role binding configuration on Helm chart for Vald Mirror Gateway
-
-To customize the cluster role configuration on the Helm chart for Vald Mirror Gateway, you may need to change the `gateway.mirror.clusterRole` configuration on the Helm chart file. The cluster role configurations are enabled by default.
-
-```yaml
-gateway:
- mirror:
- ---
- clusterRole:
- # gateway.mirror.clusterRole.enabled -- creates clusterRole resource
- enabled: true
- # gateway.mirror.clusterRole.name -- name of clusterRole
- name: gateway-mirror
- clusterRoleBinding:
- # gateway.mirror.clusterRoleBinding.enabled -- creates clusterRoleBinding resource
- enabled: true
- # gateway.mirror..clusterRoleBinding.name -- name of clusterRoleBinding
- name: gateway-mirror
- serviceAccount:
- # gateway.mirror.serviceAccount.enabled -- creates service account
- enabled: true
- # gateway.mirror.serviceAccount.name -- name of service account
- name: gateway-mirror
-```
-
--Because for example, there are a noisy problem, high memory usage for createIndex (indexing on memory), high traffic needs more memory, etc. +E.g., there are a noisy problem, high memory usage for createIndex (indexing on memory), high traffic needs more memory, etc.
-If you disable these configurations, the Vald Mirror Gateway will not work properly.
-
-
## Customize cluster role configuration on Cloud Providers
Please refer to the official guidelines to configure cluster role configuration for your cloud provider, and configure the service account name for Vald Discoverer.
diff --git a/content/docs/v1.7/user-guides/configuration.md b/content/docs/v1.7/user-guides/configuration.md
index 4de6aadb4..68a9fe512 100644
--- a/content/docs/v1.7/user-guides/configuration.md
+++ b/content/docs/v1.7/user-guides/configuration.md
@@ -1,6 +1,6 @@
---
title: "Configuration_v1.7/User Guides"
-date: 2024-08-08T10:35:17+09:00
+date: 2025-04-17T05:06:31Z
draft: false
weight: 100
description: Applies basic configuration for running Vald cluster
@@ -186,42 +186,6 @@ When the setting parameter of Vald Agent NGT is shorter than the setting value o
If this happens, the Index Manager may not function properly.
-#### Faiss
-
-Vald Agent Faiss uses [facebookresearch/faiss][faiss] as a core library for searching vectors.
-The behaviors of Faiss can be configured by setting `agent.faiss` field object.
-
-The important parameters are the followings:
-
-- `agent.faiss.dimension`
-- `agent.faiss.distance_type`
-- `agent.faiss.m`
-- `agent.faiss.metric_type`
-- `agent.faiss.nbits_per_idx`
-- `agent.faiss.nlist`
-
-Users should configure these parameters first to fit their use case.
-For further details, please read [the Faiss wiki][faiss-wiki].
-
-Vald Agent Faiss has a feature to start indexing automatically.
-The behavior of this feature can be configured with these parameters:
-
-- `agent.faiss.auto_index_duration_limit`
-- `agent.faiss.auto_index_check_duration`
-- `agent.faiss.auto_index_length`
-
-
-While the Vald Agent Faiss is in the process of creating indexes, it will ignore all search requests to the target pods.
-
-
-
-When deploying Vald Index Manager, the above parameters should be set much longer than the Vald Index Manager settings (Please refer to the Vald Index Manager section) or minus value.
-E.g., set agent.faiss.auto_index_duration_limit to "720h" or "-1h" and agent.faiss.auto_index_check_duration to "24h" or "-1h".
-This is because the Vald Index Manager accurately grasps the index information of each Vald Agent Faiss and controls the execution timing of indexing.
-When the setting parameter of Vald Agent Faiss is shorter than the setting value of Vald Index Manager, Vald Agent Faiss may start indexing by itself without the execution command from Vald Index Manager. -If this happens, the Index Manager may not function properly. -
-
#### Resource requests and limits, Pod priorities
Because the Vald Agent pod places indexes on memory, termination of agent pods causes loss of indexes.
@@ -405,7 +369,7 @@ For further details, there are references to the Helm values in the Vald GitHub
- [README of Vald Helm Chart][vald-helm-chart]
- [README of Vald-Helm-Operator Chart][vald-helm-operator-chart]
-
+
[vald-helm-chart]: https://github.com/vdaas/vald/tree/main/charts/vald
[vald-helm-operator-chart]: https://github.com/vdaas/vald/tree/main/charts/vald-helm-operator
@@ -418,5 +382,3 @@ For further details, there are references to the Helm values in the Vald GitHub
[kubernetes-topology-spread-constraints]: https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/
[yj-ngt]: https://github.com/yahoojapan/NGT
[yj-ngt-wiki]: https://github.com/yahoojapan/NGT/wiki
-[faiss]: https://github.com/facebookresearch/faiss
-[faiss-wiki]: https://github.com/facebookresearch/faiss/wiki
diff --git a/content/docs/v1.7/user-guides/deployment.md b/content/docs/v1.7/user-guides/deployment.md
index 43c478b0a..1a6f5d670 100644
--- a/content/docs/v1.7/user-guides/deployment.md
+++ b/content/docs/v1.7/user-guides/deployment.md
@@ -1,6 +1,6 @@
---
title: "Deployment_v1.7/User Guides"
-date: 2024-08-08T10:35:17+09:00
+date: 2025-04-17T05:06:32Z
draft: false
weight: 600
description: How to launch Vald cluster on your Kubernetes cluster
@@ -38,7 +38,7 @@ If Helm is not installed, please install [Helm](https://helm.sh/docs/intro/insta
-E.g., set agent.faiss.auto_index_duration_limit to "720h" or "-1h" and agent.faiss.auto_index_check_duration to "24h" or "-1h".
-This is because the Vald Index Manager accurately grasps the index information of each Vald Agent Faiss and controls the execution timing of indexing.
-When the setting parameter of Vald Agent Faiss is shorter than the setting value of Vald Index Manager, Vald Agent Faiss may start indexing by itself without the execution command from Vald Index Manager. -If this happens, the Index Manager may not function properly. -
Installation command for Helm
```bash -curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash +curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash ```
OpenTelemetry Collector handles receiving / processing / exporting the telemetry data, which does NOT depend on the vendors' format.
-All Vald components can send OTLP-compliant telemetry data, like metrics, traces, or logs, to the OpenTelemetry Collector.
+All Vald components can send OTLP-compliant telemetry data, like metrics, traces, or logs, to the OpenTelemerty Collector.
The telemetry data is used to monitor or visualize with the observing tools.
## Build an observing environment
@@ -34,7 +34,7 @@ The recommended observability environment consists of the following:
- Jaeger Operator and Jaeger
- Prometheus Operator and Prometheus
- Grafana
-- OpenTelemetry Operator and OpenTelemetry Collector
+- OpenTelemetry Operator and OpenTelemtry Collector
Vald provides [the default manifests](https://github.com/vdaas/vald/tree/main/k8s/metrics) and [the make commands](https://github.com/vdaas/vald/blob/main/Makefile.d/k8s.mk) for deploying those components.
Please refer to the following sections to deploy each component.
@@ -78,7 +78,7 @@ make k8s/metrics/grafana/deploy
### OpenTelemetry Operator and OpenTelemetry Collector
Vald uses OpenTelemetry Collector to get the telemetry data and export it to monitor backend components.
-The following command deploys OpenTelemetry Collector via OpenTelemetry Collector.
+The following command deploys OpenTelemtry Collector via OpenTelemerty Collector.
Before executing the following command, please ensure the Prometheus operator runs healthy.
@@ -143,7 +143,7 @@ defaults:
- "go_version"
- "go_os"
- "go_arch"
- - "algorithm_info"
+ - "ngt_version"
# enable memory metrics
enable_memory: true
# enable goroutine metrics
@@ -187,7 +187,7 @@ defaults:
#### Specify the Telemetry attribute
-You can add the component information to the attribute of telemetry data by editing `defaults.observability.otlp.attribute`.
+You can add the component information to the attribute of telemetry data by editing `defaults.observability.otlp.attirbute`.
E.g., when setting `vald-agent-ngt-0` as `agent.observability.otlp.attribute.pod_name`, `target_pod: vald-agent-ngt-0` will be added to the attribute.
These attributes are set auto by the environment values, so Vald recommends using default values unless there is a specific reason.
diff --git a/content/docs/v1.7/user-guides/read-replica-and-rotator.md b/content/docs/v1.7/user-guides/read-replica-and-rotator.md
deleted file mode 100644
index f053d92cd..000000000
--- a/content/docs/v1.7/user-guides/read-replica-and-rotator.md
+++ /dev/null
@@ -1,116 +0,0 @@
----
-title: "Read Replica and Rotator_v1.7/User Guides"
-date: 2024-08-08T10:35:17+09:00
-draft: false
-weight: 1500
-description: How to improve search request speed
-menu:
- userguides:
- parent: User Guides
----
-
-# Read Replica and Rotator
-
-Read replica enhances the search QPS (Queries Per Second) of the Vald cluster by deploying read-only agents in addition to the regular agents and distributing the requests among them. Read replica is deployed as Kubernetes deployments, and depending on the number of replicas (N), QPS increases by approximately 1.7 to 1.8 times \* N.
-
-
- ```
-
-## Architecture
-
-Read replica mainly consists of the following four parts.
-
-
-
-### Read replica deployment
-
-The deployment that generates Pods where the actual processing of read replica takes place. Read replica accepts read requests (search) and reads the index from the read replica PVC.
-
-### Read replica PVC
-
-The PVC for read replica Pods is used to read the index. It is generated based on the latest snapshot from the PVC of the regular agent. Unlike the agent PVC, it is generated as ROX, allowing it to be read from multiple Pods.
-
-### Index operator
-
-The operator handles the following processes:
-
-1. Monitoring the time when the agent saved the index to the PVC and when the read replica performed index rotation
-1. Generating [Read replica rotator](#read-replica-rotator) job when an index save occurs after the most recent rotation
-
-> The Index operator also manages the timing of index create/save operations other than those mentioned above. Please refer to another document for details.
-
-### Read replica rotator
-
-The Kubernetes job handles the following processes:
-
-1. Creating a snapshot from the agent's PVC
-1. Generating a PVC for read replica from the snapshot
-1. Rolling update of the read replica deployment to launch a group of read replica pods with the latest index
-
-## Important notes
-
-Result consistency is guaranteed
-
-There is a time lag between index insertion, agent save, and the completion of read replica rotation. During this time, there may be inconsistencies between the index in the agent itself and the index in the read replica.
-
-- Sufficient infrastructure is required for QPS scaling
-
- Even if read replicas are deployed, QPS will not scale if sufficient resources are not available in the Kubernetes cluster. Specifically, agent resources and read replica resources should be deployed on separate nodes. Vald sets `podAntiAffinity` to ensure that agent resources and read replica resources are deployed on separate nodes as much as possible.
diff --git a/content/docs/v1.7/user-guides/sdks.md b/content/docs/v1.7/user-guides/sdks.md
index dc67f0fe7..4ab0d6fdd 100644
--- a/content/docs/v1.7/user-guides/sdks.md
+++ b/content/docs/v1.7/user-guides/sdks.md
@@ -1,6 +1,6 @@
---
title: "Sdks_v1.7/User Guides"
-date: 2024-12-06T15:24:21+09:00
+date: 2025-04-17T05:06:32Z
draft: false
weight: 1400
description: Sends request and receives response from the Vald cluster
@@ -27,6 +27,7 @@ Here is the list of Official SDKs.
- [Java](https://github.com/vdaas/vald-client-java)
- [Python](https://github.com/vdaas/vald-client-python)
- [Node.js](https://github.com/vdaas/vald-client-node)
+- [Clojure](https://github.com/vdaas/vald-client-clj)
## How to use SDKs
diff --git a/content/docs/v1.7/user-guides/upgrade-cluster.md b/content/docs/v1.7/user-guides/upgrade-cluster.md
index 88acd41ee..e0bcbe12e 100644
--- a/content/docs/v1.7/user-guides/upgrade-cluster.md
+++ b/content/docs/v1.7/user-guides/upgrade-cluster.md
@@ -1,6 +1,6 @@
---
title: "Upgrade Cluster_v1.7/User Guides"
-date: 2024-08-08T10:35:17+09:00
+date: 2025-04-17T05:06:32Z
draft: false
weight: 800
description: Describes hot to upgrade Vald version for the running Vald cluster
@@ -38,7 +38,7 @@ You can edit your `values.yaml` and update your Vald cluster by following steps:
```bash
helm upgrade vald/vald --values
- # e.g., helm upgrade vald vald/vald --values values.yaml
+ # e.g., helm upgrade vald vald/vald --values valeus.yaml
```
### Using vald-helm-operator
@@ -118,7 +118,7 @@ If there is major or minor upgrading, the chart structure may have changed.
```bash
helm upgrade vald/vald --values
- # e.g., helm upgrade vald vald/vald --values values.yaml
+ # e.g., helm upgrade vald vald/vald --values valeus.yaml
```
### Using vald-helm-operator
diff --git a/stage b/stage
index a945dc893..8b69f1e9c 160000
--- a/stage
+++ b/stage
@@ -1 +1 @@
-Subproject commit a945dc893328d8fc988f2cdbd5ed19dadbfbbd1b
+Subproject commit 8b69f1e9cbb3585c2c697716fdd20c54a1ebf74f
-The increase in QPS is possible with sufficient infrastructure (see Important notes).
-
-
-## How to deploy read replica
-
-The read replica is managed with a separate chart from the Vald cluster and is deployed as an addon to the Vald cluster. The Vald cluster should be deployed first, followed by the deployment of the read replica.
-
-> The reason Vald and Vald-readreplica are in separate charts is to avoid conflicts between the read replica's restart and the Helm operator's processes when Vald is managed by a helm operator. Therefore, the read replica will always be deployed using Helm commands.
-
-### When you deploy Vald with Helm command
-
-1. Edit `values.yaml` like below (Please refer to [deployment](/docs/v1.7/user-guides/deployment) for other fields.)
-
- ```yaml
- agent:
- ngt:
- export_index_info_to_k8s: true
- readreplica:
- enabled: true
- minReplicas: 1 # if you don't use hpa, this will be the replicas of the Deployment
- maxReplicas: 3
- hpa:
- enabled: true # if you prefer to use hpa
- targetCPUUtilizationPercentage: 80
- manager:
- index:
- operator:
- enabled: true
- rotation_job_concurrency: 2
- ```
-
-1. Deploy Vald cluster
-
- ```bash
- helm install vald vald/vald --values values.yaml
- ```
-
-1. Deploy `vald-readreplica` with the same `values.yaml`
-
- ```bash
- helm install vald-readreplica vald/vald-readreplica --values values.yaml
- ```
-
-### When you deploy Vald cluster with `vald-helm-operator`
-
-1. Edit `valdrelease.yaml` with the same fields as above
-
-1. Deploy Vald cluster
-
- ```bash
- helm install vald-helm-operator-release vald/vald-helm-operator
- kubectl apply -f valdrelease.yaml
- ```
-
-1. Deploy `vald-readreplica`
-
- ```bash
- helm install vald-readreplica vald/vald-readreplica --values
-
-### Read replica deployment
-
-The deployment that generates Pods where the actual processing of read replica takes place. Read replica accepts read requests (search) and reads the index from the read replica PVC.
-
-### Read replica PVC
-
-The PVC for read replica Pods is used to read the index. It is generated based on the latest snapshot from the PVC of the regular agent. Unlike the agent PVC, it is generated as ROX, allowing it to be read from multiple Pods.
-
-### Index operator
-
-The operator handles the following processes:
-
-1. Monitoring the time when the agent saved the index to the PVC and when the read replica performed index rotation
-1. Generating [Read replica rotator](#read-replica-rotator) job when an index save occurs after the most recent rotation
-
-> The Index operator also manages the timing of index create/save operations other than those mentioned above. Please refer to another document for details.
-
-### Read replica rotator
-
-The Kubernetes job handles the following processes:
-
-1. Creating a snapshot from the agent's PVC
-1. Generating a PVC for read replica from the snapshot
-1. Rolling update of the read replica deployment to launch a group of read replica pods with the latest index
-
-## Important notes
-
-Result consistency is guaranteed
-
-There is a time lag between index insertion, agent save, and the completion of read replica rotation. During this time, there may be inconsistencies between the index in the agent itself and the index in the read replica.
-
-- Sufficient infrastructure is required for QPS scaling
-
- Even if read replicas are deployed, QPS will not scale if sufficient resources are not available in the Kubernetes cluster. Specifically, agent resources and read replica resources should be deployed on separate nodes. Vald sets `podAntiAffinity` to ensure that agent resources and read replica resources are deployed on separate nodes as much as possible.
diff --git a/content/docs/v1.7/user-guides/sdks.md b/content/docs/v1.7/user-guides/sdks.md
index dc67f0fe7..4ab0d6fdd 100644
--- a/content/docs/v1.7/user-guides/sdks.md
+++ b/content/docs/v1.7/user-guides/sdks.md
@@ -1,6 +1,6 @@
---
title: "Sdks_v1.7/User Guides"
-date: 2024-12-06T15:24:21+09:00
+date: 2025-04-17T05:06:32Z
draft: false
weight: 1400
description: Sends request and receives response from the Vald cluster
@@ -27,6 +27,7 @@ Here is the list of Official SDKs.
- [Java](https://github.com/vdaas/vald-client-java)
- [Python](https://github.com/vdaas/vald-client-python)
- [Node.js](https://github.com/vdaas/vald-client-node)
+- [Clojure](https://github.com/vdaas/vald-client-clj)
## How to use SDKs
diff --git a/content/docs/v1.7/user-guides/upgrade-cluster.md b/content/docs/v1.7/user-guides/upgrade-cluster.md
index 88acd41ee..e0bcbe12e 100644
--- a/content/docs/v1.7/user-guides/upgrade-cluster.md
+++ b/content/docs/v1.7/user-guides/upgrade-cluster.md
@@ -1,6 +1,6 @@
---
title: "Upgrade Cluster_v1.7/User Guides"
-date: 2024-08-08T10:35:17+09:00
+date: 2025-04-17T05:06:32Z
draft: false
weight: 800
description: Describes hot to upgrade Vald version for the running Vald cluster
@@ -38,7 +38,7 @@ You can edit your `values.yaml` and update your Vald cluster by following steps:
```bash
helm upgrade