v1.13: Add remote federated search documentation (#3144)

---------

Co-authored-by: Louis Dureuil <[email protected]>

.code-samples.meilisearch.yaml

@@ -1444,3 +1444,46 @@ experimental_post_logs_stream_1: |-
 experimental_delete_logs_stream_1: |-
   curl \
     -X DELETE MEILISEARCH_URL/logs/stream
+multi_search_remote_federated_1: |-
+  curl \
+    -X POST 'MEILISEARCH_URL/multi-search' \
+    -H 'Content-Type: application/json' \
+    --data-binary '{
+      "federation": {},
+      "queries": [
+        {
+          "indexUid": "movies",
+          "q": "batman",
+          "federationOptions": {
+            "remote": "ms-00"
+          }
+        },
+        {
+          "indexUid": "movies",
+          "q": "batman",
+          "federationOptions": {
+            "remote": "ms-01"
+          }
+        }
+      ]
+    }'
+get_network_1: |-
+  curl \
+    -X GET 'MEILISEARCH_URL/network'
+update_network_1: |-
+  curl \
+    -X PATCH 'MEILISEARCH_URL/network' \
+    -H 'Content-Type: application/json' \
+    --data-binary '{
+      "self": "ms-00",
+      "remotes": {
+        "ms-00": {
+          "url": "http://INSTANCE_URL",
+          "searchApiKey": "INSTANCE_API_KEY"
+        },
+        "ms-01": {
+          "url": "http://ANOTHER_INSTANCE_URL",
+          "searchApiKey": "ANOTHER_INSTANCE_API_KEY"
+        }
+      }
+    }'

config/sidebar-learn.json

@@ -267,6 +267,11 @@
         "label": "Using multi-search to perform a federated search",
         "slug": "performing_federated_search"
       },
+      {
+        "source": "learn/multi_search/implement_sharding.mdx",
+        "label": "Implement sharding with remote federated search",
+        "slug": "implement_sharding"
+      },
       {
         "source": "learn/multi_search/multi_search_vs_federated_search.mdx",
         "label": "Differences between multi-search and federated search",

config/sidebar-reference.json

@@ -28,6 +28,11 @@
         "label": "Multi-search",
         "slug": "multi_search"
       },
+      {
+        "source": "reference/api/network.mdx",
+        "label": "Network",
+        "slug": "network"
+      },
       {
         "source": "reference/api/similar.mdx",
         "label": "Similar documents",

learn/multi_search/implement_sharding.mdx

@@ -0,0 +1,129 @@
+---
+title: Implement sharding with remote federated search — Meilisearch documentation
+description: This guide walks you through implementing a sharding strategy by activating the `/network` route, configuring the network object, and performing remote federated searches.
+---
+
+# Implement sharding with remote federated search <NoticeTag type="experimental" label="experimental" />
+
+Sharding is the process of splitting an index containing many documents into multiple smaller indexes, often called shards. This horizontal scaling technique is useful when handling large databases. In Meilisearch, the best way to implement a sharding strategy is to use remote federated search.
+
+This guide walks you through activating the `/network` route, configuring the network object, and performing remote federated searches.
+
+<Capsule intent="tip" title="Configuring multiple instances">
+To minimize issues and limit unexpected behavior, instance, network, and index configuration should be identical for all shards. This guide describes the individual steps you must take on a single instance and assumes you will replicate them across all instances.
+</Capsule>
+
+## Prerequisites
+
+- Multiple Meilisearch projects (instances) running Meilisearch >=v1.13
+
+## Activate the `/network` endpoint
+
+### Meilisearch Cloud
+
+If you are using Meilisearch Cloud, contact support to enable this feature in your projects.
+
+### Self-hosting
+
+Use the `/experimental-features` route to enable `network`:
+
+```sh
+curl \
+  -X PATCH 'MEILISEARCH_URL/experimental-features/' \
+  -H 'Content-Type: application/json'  \
+  --data-binary '{
+    "network": true
+  }'
+```
+
+Meilisearch should respond immediately, confirming the route is now accessible. Repeat this process for all instances.
+
+## Configuring the network object
+
+Next, you must configure the network object. It consists of the following fields:
+
+- `remotes`: defines a list with the required information to access each remote instance
+- `self`: specifies which of the configured `remotes` corresponds to the current instance
+
+### Setting up the list of remotes
+
+Use the `/network` route to configure the `remotes` field of the network object. `remotes` should be an object containing one or more objects. Each one of the nested objects should consist of the name of each instance, associated with its URL and an API key with search permission:
+
+```sh
+curl \
+  -X PATCH 'MEILISEARCH_URL/network' \
+  -H 'Content-Type: application/json' \
+  --data-binary '{
+    "remotes": {
+      "REMOTE_NAME_1": {
+        "url": "INSTANCE_URL_1",
+        "searchApiKey": "SEARCH_API_KEY_1"
+      },
+      "REMOTE_NAME_2": {
+        "url": "INSTANCE_URL_2",
+        "searchApiKey": "SEARCH_API_KEY_2"
+      },
+      "REMOTE_NAME_3": {
+        "url": "INSTANCE_URL_3",
+        "searchApiKey": "SEARCH_API_KEY_3"
+      },
+      …
+    }
+  }'
+```
+
+Configure the entire set of remote instances in your sharded database, making sure to send the same remotes to each instance.
+
+### Specify the name of the current instance
+
+Now all instances share the same list of remotes, set the `self` field to specify which of the remotes corresponds to the current instance:
+
+```sh
+curl \
+  -X PATCH 'MEILISEARCH_URL/network' \
+  -H 'Content-Type: application/json' \
+  --data-binary '{
+    "self": "REMOTE_NAME_1"
+  }'
+```
+
+Meilisearch processes searches on the remote that corresponds to `self` locally instead of making a remote request.
+
+### Adding or removing an instance
+
+Changing the topology of the network involves moving some documents from an instance to another, depending on your hashing scheme.
+
+As Meilisearch does not provide atomicity across multiple instances, you will need to either:
+
+1. accept search downtime while migrating documents
+2. accept some documents will not appear in search results during the migration
+3. accept some duplicate documents may appear in search results during the migration
+
+#### Reducing downtime
+
+If your disk space allows, you can reduce the downtime by applying the following algorithm:
+
+1. Create a new temporary index in each remote instance
+2. Compute the new instance for each document
+3. Send the documents to the temporary index of their new instance
+4. Once Meilisearch has copied all documents to their instance of destination, swap the new index with the previously used index
+5. Delete the temporary index after the swap
+6. Update network configuration and search queries across all instances
+
+## Create indexes and add documents
+
+Create the same empty indexes with the same settings on all instances. Keeping the settings and indexes in sync is important to avoid errors and unexpected behavior, though not strictly required.
+
+Distribute your documents across all instances. Do not send the same document to multiple instances as this may lead to duplicate search results. Similarly, you should ensure all future versions of a document are sent to the same instance. Meilisearch recommends you hash their primary key using [rendezvous hashing](https://en.wikipedia.org/wiki/Rendezvous_hashing).
+
+### Updating index settings
+
+Changing settings in a sharded database is not fundamentally different from changing settings on a single Meilisearch instance. If the update enables a feature, such as setting filterable attributes, wait until all changes have been processed before using the `filter` search parameter in a query. Likewise, if an update disables a feature, first remove it from your search requests, then update your settings.
+
+## Perform a search
+
+Send your federated search request containing one query per instance:
+
+<CodeSamples id="multi_search_remote_federated_1" />
+
+If all instances share the same network configuration, you can send the search request to any instance. Having `"remote": "ms-00"` appear in the list of queries on the instance of that name will not cause an actual proxy search thanks to `network.self`.

learn/resources/experimental_features_overview.mdx

@@ -42,13 +42,15 @@ Activating or deactivating experimental features this way does not require you t
 
 ## Current experimental features
 
-| Name                                                                             | Description                                                | How to configure                                  |
-|----------------------------------------------------------------------------------|------------------------------------------------------------|---------------------------------------------------|
-| [Limit task batch size](/learn/self_hosted/configure_meilisearch_at_launch)         | Limits number of tasks processed in a single batch         | At launch with a CLI flag or environment variable |
-| [Log customization](/reference/api/logs)                       | Customize log output and set up log streams                | At launch with a CLI flag or environment variable, during runtime with the API route |
-| [Metrics API](/reference/api/metrics)                                       | Exposes Prometheus-compatible analytics data               | At launch with a CLI flag or environment variable, during runtime with the API route |
-| [Reduce indexing memory usage](/learn/self_hosted/configure_meilisearch_at_launch) | Optimizes indexing performance                             | At launch with a CLI flag or environment variable |
-| [Replication parameters](/learn/self_hosted/configure_meilisearch_at_launch)             | Alters task processing for clustering compatibility        | At launch with a CLI flag or environment variable |
-| [Search queue size](/learn/self_hosted/configure_meilisearch_at_launch)                       | Configure maximum number of concurrent search requests     | At launch with a CLI flag or environment variable |
-| [`CONTAINS` filter operator](/learn/filtering_and_sorting/filter_expression_reference#contains)                               | Enables usage of `CONTAINS` with the `filter` search parameter | During runtime with the API route                 |
-| [Edit documents with function](/reference/api/documents#update-documents-with-function)                               | Use a RHAI function to edit documents directly in the Meilisearch database | During runtime with the API route                 |
+| Name                                                                                            | Description                                                                | How to configure                            |
+| ----------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | ------------------------------------------- |
+| [Limit task batch size](/learn/self_hosted/configure_meilisearch_at_launch)                     | Limits number of tasks processed in a single batch                         | CLI flag or environment variable            |
+| [Log customization](/reference/api/logs)                                                        | Customize log output and set up log streams                                | CLI flag or environment variable, API route |
+| [Metrics API](/reference/api/metrics)                                                           | Exposes Prometheus-compatible analytics data                               | CLI flag or environment variable, API route |
+| [Reduce indexing memory usage](/learn/self_hosted/configure_meilisearch_at_launch)              | Optimizes indexing performance                                             | CLI flag or environment variable            |
+| [Replication parameters](/learn/self_hosted/configure_meilisearch_at_launch)                    | Alters task processing for clustering compatibility                        | CLI flag or environment variable            |
+| [Search queue size](/learn/self_hosted/configure_meilisearch_at_launch)                         | Configure maximum number of concurrent search requests                     | CLI flag or environment variable            |
+| [`CONTAINS` filter operator](/learn/filtering_and_sorting/filter_expression_reference#contains) | Enables usage of `CONTAINS` with the `filter` search parameter             | API route                                   |
+| [Edit documents with function](/reference/api/documents#update-documents-with-function)         | Use a RHAI function to edit documents directly in the Meilisearch database | API route                                   |
+| [`/network` route](/reference/api/network)                                                      | Enable `/network` route                                                    | API route                                   |
+| [Dumpless upgrade](/learn/self_hosted/configure_meilisearch_at_launch#dumpless-upgrade)         | Upgrade Meilisearch without generating a dump                              | API route                                   |

reference/api/experimental_features.mdx

@@ -20,16 +20,18 @@ The experimental API route is not compatible with all experimental features. Con
   "metrics": false,
   "logsRoute": true,
   "containsFilter": false,
-  "editDocumentsByFunction": false
+  "editDocumentsByFunction": false,
+  "network": false
 }
 ```
 
-| Name                          | Type    | Description                                                               |
+| Name                          | Type    | Description                                    |
 | :---------------------------- | :------ | :--------------------------------------------- |
 | **`metrics`**                 | Boolean | `true` if feature is active, `false` otherwise |
 | **`logsRoute`**               | Boolean | `true` if feature is active, `false` otherwise |
 | **`containsFilter`**          | Boolean | `true` if feature is active, `false` otherwise |
 | **`editDocumentsByFunction`** | Boolean | `true` if feature is active, `false` otherwise |
+| **`network`**                 | Boolean | `true` if feature is active, `false` otherwise |
 
 ## Get all experimental features
 
@@ -48,7 +50,8 @@ Get a list of all experimental features that can be activated via the `/experime
   "metrics": false,
   "logsRoute": true,
   "containsFilter": false,
-  "editDocumentsByFunction": false
+  "editDocumentsByFunction": false,
+  "network": false
 }
 ```
 
@@ -75,6 +78,7 @@ Setting a field to `null` leaves its value unchanged.
   "metrics": false,
   "logsRoute": true,
   "containsFilter": false,
-  "editDocumentsByFunction": false
+  "editDocumentsByFunction": false,
+  "network": false
 }
 ```

reference/api/keys.mdx

@@ -101,6 +101,8 @@ For security reasons, we do not recommend creating keys that can perform all act
 | **`keys.create`**      | Provides access to the [create key](#create-a-key) endpoint                                                                                                                                                                                                                                                                  |
 | **`keys.update`**      | Provides access to the [update key](#update-a-key) endpoint                                                                                                                                                                                                                                                                  |
 | **`keys.delete`**      | Provides access to the [delete key](#delete-a-key) endpoint                                                                                                                                                                                                                                                                  |
+| **`network.get`**      | Provides access to the [get the network object](/reference/api/network#get-the-network-object) endpoint                                                                                                                                                                                                                                                                |
+| **`network.update`**   | Provides access to the [update the network object](/reference/api/network#update-the-network-object) endpoint                                                                                                                                                                                                                                                                  |
 
 ### `indexes`