|
| 1 | +--- |
| 2 | +categories: |
| 3 | +- docs |
| 4 | +- develop |
| 5 | +- stack |
| 6 | +- oss |
| 7 | +- rs |
| 8 | +- rc |
| 9 | +- oss |
| 10 | +- kubernetes |
| 11 | +- clients |
| 12 | +description: Index and query embeddings with Redis vector sets |
| 13 | +linkTitle: Vector set embeddings |
| 14 | +title: Vector set embeddings |
| 15 | +weight: 35 |
| 16 | +bannerText: Vector set is a new data type that is currently in preview and may be subject to change. |
| 17 | +bannerChildren: true |
| 18 | +--- |
| 19 | + |
| 20 | +A Redis [vector set]({{< relref "/develop/data-types/vector-sets" >}}) lets |
| 21 | +you store a set of unique keys, each with its own associated vector. |
| 22 | +You can then retrieve keys from the set according to the similarity between |
| 23 | +their stored vectors and a query vector that you specify. |
| 24 | + |
| 25 | +You can use vector sets to store any type of numeric vector but they are |
| 26 | +particularly optimized to work with text embedding vectors (see |
| 27 | +[Redis for AI]({{< relref "/develop/ai" >}}) to learn more about text |
| 28 | +embeddings). The example below shows how to use the |
| 29 | +[`Hugot`](https://github.com/knights-analytics/hugot) |
| 30 | +library to generate vector embeddings and then |
| 31 | +store and retrieve them using a vector set with `go-redis`. |
| 32 | + |
| 33 | +## Initialize |
| 34 | + |
| 35 | +Start by [installing]({{< relref "/develop/clients/go#install" >}}) `go-redis` if you haven't already done so. Note that you need `go-redis` |
| 36 | +[v9.10.0](https://github.com/redis/go-redis/releases/tag/v9.10.0) |
| 37 | +or later to use vector sets. |
| 38 | + |
| 39 | +Also, install `hugot`: |
| 40 | + |
| 41 | +```bash |
| 42 | +go get github.com/knights-analytics/hugot |
| 43 | +``` |
| 44 | + |
| 45 | +In a new Go file, add the required imports: |
| 46 | + |
| 47 | +{{< clients-example set="home_vecsets" step="import" lang_filter="Go" >}} |
| 48 | +{{< /clients-example >}} |
| 49 | + |
| 50 | +These include the `Hugot` library to generate an embedding from a section of text. |
| 51 | +This example uses an instance of the `SentenceTransformer` model |
| 52 | +[`all-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) |
| 53 | +for the embeddings. This model generates vectors with 384 dimensions, regardless |
| 54 | +of the length of the input text, but note that the input is truncated to 256 |
| 55 | +tokens (see |
| 56 | +[Word piece tokenization](https://huggingface.co/learn/nlp-course/en/chapter6/6) |
| 57 | +at the [Hugging Face](https://huggingface.co/) docs to learn more about the way tokens |
| 58 | +are related to the original text). |
| 59 | + |
| 60 | +{{< clients-example set="home_vecsets" step="model" lang_filter="Go" >}} |
| 61 | +{{< /clients-example >}} |
| 62 | + |
| 63 | +## Create the data |
| 64 | + |
| 65 | +The example data is contained in a map with some brief |
| 66 | +descriptions of famous people: |
| 67 | + |
| 68 | +{{< clients-example set="home_vecsets" step="data" lang_filter="Go" >}} |
| 69 | +{{< /clients-example >}} |
| 70 | + |
| 71 | +## Add the data to a vector set |
| 72 | + |
| 73 | +The next step is to connect to Redis and add the data to a new vector set. |
| 74 | + |
| 75 | +The code below iterates through the `peopleData` map and adds corresponding |
| 76 | +elements to a vector set called `famousPeople`. |
| 77 | + |
| 78 | +Use the |
| 79 | +`RunPipeline()` method of `embeddingPipeline` to generate the |
| 80 | +embedding as an array of `float32` values, then use a loop like the one |
| 81 | +shown below to convert the `float32` array to a `float64` array. |
| 82 | +You can then pass this array to the |
| 83 | +[`VAdd()`]({{< relref "/commands/vadd" >}}) command to set the embedding. |
| 84 | + |
| 85 | +The call to `VAdd()` also adds the `born` and `died` values from the |
| 86 | +original map as attribute data. You can access this during a query |
| 87 | +or by using the [`VGetAttr()`]({{< relref "/commands/vgetattr" >}}) method. |
| 88 | + |
| 89 | +{{< clients-example set="home_vecsets" step="add_data" lang_filter="Go" >}} |
| 90 | +{{< /clients-example >}} |
| 91 | + |
| 92 | +## Query the vector set |
| 93 | + |
| 94 | +You can now query the data in the set. The basic approach is to use the |
| 95 | +`RunPipeline()` method to generate another embedding vector for the query text. |
| 96 | +(This is the same method used to add the elements to the set.) Then, pass |
| 97 | +the query vector to [`VSim()`]({{< relref "/commands/vsim" >}}) to return elements |
| 98 | +of the set, ranked in order of similarity to the query. |
| 99 | + |
| 100 | +Start with a simple query for "actors": |
| 101 | + |
| 102 | +{{< clients-example set="home_vecsets" step="basic_query" lang_filter="Go" >}} |
| 103 | +{{< /clients-example >}} |
| 104 | + |
| 105 | +This returns the following list of elements (formatted slightly for clarity): |
| 106 | + |
| 107 | +``` |
| 108 | +'actors': Masako Natsume, Chaim Topol, Linus Pauling, Marie Fredriksson, |
| 109 | + Maryam Mirzakhani, Marie Curie, Freddie Mercury, Paul Erdos |
| 110 | +``` |
| 111 | + |
| 112 | +The first two people in the list are the two actors, as expected, but none of the |
| 113 | +people from Linus Pauling onward was especially well-known for acting (and there certainly |
| 114 | +isn't any information about that in the short description text). |
| 115 | +As it stands, the search attempts to rank all the elements in the set, based |
| 116 | +on the information contained in the embedding model. |
| 117 | +You can use the `Count` parameter of `VSimWithArgs()` to limit the list of elements |
| 118 | +to just the most relevant few items: |
| 119 | + |
| 120 | +{{< clients-example set="home_vecsets" step="limited_query" lang_filter="Go" >}} |
| 121 | +{{< /clients-example >}} |
| 122 | + |
| 123 | +The reason for using text embeddings rather than simple text search |
| 124 | +is that the embeddings represent semantic information. This allows a query |
| 125 | +to find elements with a similar meaning even if the text is |
| 126 | +different. For example, the word "entertainer" doesn't appear in any of the |
| 127 | +descriptions but if you use it as a query, the actors and musicians are ranked |
| 128 | +highest in the results list: |
| 129 | + |
| 130 | +{{< clients-example set="home_vecsets" step="entertainer_query" lang_filter="Go" >}} |
| 131 | +{{< /clients-example >}} |
| 132 | + |
| 133 | +Similarly, if you use "science" as a query, you get the following results: |
| 134 | + |
| 135 | +``` |
| 136 | +'science': Marie Curie, Linus Pauling, Maryam Mirzakhani, Paul Erdos, |
| 137 | +Marie Fredriksson, Freddie Mercury, Masako Natsume, Chaim Topol |
| 138 | +``` |
| 139 | + |
| 140 | +The scientists are ranked highest but they are then followed by the |
| 141 | +mathematicians. This seems reasonable given the connection between mathematics |
| 142 | +and science. |
| 143 | + |
| 144 | +You can also use |
| 145 | +[filter expressions]({{< relref "/develop/data-types/vector-sets/filtered-search" >}}) |
| 146 | +with `VSimWithArgs()` to restrict the search further. For example, |
| 147 | +repeat the "science" query, but this time limit the results to people |
| 148 | +who died before the year 2000: |
| 149 | + |
| 150 | +{{< clients-example set="home_vecsets" step="filtered_query" lang_filter="Go" >}} |
| 151 | +{{< /clients-example >}} |
| 152 | + |
| 153 | +Note that the boolean filter expression is applied to items in the list |
| 154 | +before the vector distance calculation is performed. Items that don't |
| 155 | +pass the filter test are removed from the results completely, rather |
| 156 | +than just reduced in rank. This can help to improve the performance of the |
| 157 | +search because there is no need to calculate the vector distance for |
| 158 | +elements that have already been filtered out of the search. |
| 159 | + |
| 160 | +## More information |
| 161 | + |
| 162 | +See the [vector sets]({{< relref "/develop/data-types/vector-sets" >}}) |
| 163 | +docs for more information and code examples. See the |
| 164 | +[Redis for AI]({{< relref "/develop/ai" >}}) section for more details |
| 165 | +about text embeddings and other AI techniques you can use with Redis. |
| 166 | + |
| 167 | +You may also be interested in |
| 168 | +[vector search]({{< relref "/develop/clients/go/vecsearch" >}}). |
| 169 | +This is a feature of the |
| 170 | +[Redis query engine]({{< relref "/develop/ai/search-and-query" >}}) |
| 171 | +that lets you retrieve |
| 172 | +[JSON]({{< relref "/develop/data-types/json" >}}) and |
| 173 | +[hash]({{< relref "/develop/data-types/hashes" >}}) documents based on |
| 174 | +vector data stored in their fields. |
0 commit comments