[DOCS-1930] Add generic user metadata documentation (#311)

_config.yml

@@ -335,3 +335,18 @@ monitoring:
 
 changeNotify:
   useRESTnotSMB: Qumulo provides REST access to the same [change notification system](how-file-system-change-notifications-work.html) that backs SMB2 `CHANGE_NOTIFY`. Because the notification interface and types are easier to work with compared to SMB2, we recommend programming by using REST rather than SMB `CHANGE_NOTIFY`.
+
+udm:
+  t: User-Defined Metadata
+  s: User-defined metadata
+  l: user-defined metadata
+  hex: To specify a non-text value for the user-defined metadata, use the <code>--base64-value</code> or <code>--hex-value</code> flag.
+  snap: To access the user-defined metadata within a file snapshot, use the <code>--snapshot</code> flag and specify the shapshot ID.
+
+guides:
+  anq: Azure Native Qumulo Administrator Guide
+  onprem: Qumulo On-Premises Administrator Guide
+  alert: Qumulo Alerts Administrator Guide
+  cli: Qumulo <code>qq</code> CLI Command Guide
+  hard: Qumulo-Certified Hardware Servicing Guide
+  integ: Qumulo Integration Guide

_data/sidebars/administrator_guide_sidebar.yml

@@ -160,6 +160,15 @@ entries:
       url: /administrator-guide/qq-cli/enabling-autocomplete.html
       output: web, pdf
 
+  - title: Metadata
+    url: /administrator-guide/metadata/index.html
+    output: web, pdf
+    folderitems:
+
+    - title: Managing User-Defined Metadata
+      url: /administrator-guide/metadata/managing-user-defined-metadata.html
+      output: web, pdf
+
   - title: Snapshots
     url: /administrator-guide/snapshots/index.html
     output: web, pdf

_data/sidebars/azure_guide_sidebar.yml

@@ -144,6 +144,15 @@ entries:
     - title: Enabling Autocomplete for the qq CLI
       url: /azure-administrator-guide/qq-cli/enabling-autocomplete.html
       output: web, pdf
+
+  - title: Metadata
+    url: /azure-administrator-guide/metadata/index.html
+    output: web, pdf
+    folderitems:
+
+    - title: Managing User-Defined Metadata
+      url: /azure-administrator-guide/metadata/managing-user-defined-metadata.html
+      output: web, pdf
 
   - title: Snapshots
     url: /azure-administrator-guide/snapshots/index.html

_includes/content-reuse/admin-guides/metadata/managing-user-defined-metadata.md

@@ -0,0 +1,107 @@
+## How {{site.udm.t}} Works in Qumulo Core
+
+Qumulo Core lets you add _{{site.udm.l}}_ to any file type stored in its file system. {{site.udm.s}} comprises user-specified key-value pairs that have the following requirements:
+
+* The key must be a Unicode string.
+* The value must be a sequence of bytes.
+* The total size of each key-value pair must be under 400 KB.
+
+
+<a id="keyspace"></a>
+## Keyspace Types and Functions
+{{site.udm.s}} in Qumulo Core is divided into `GENERIC` and `S3` _keyspaces_. Keyspaces work like _containers_ for key-value pairs. The `S3` keyspace primarily supports the [S3 API](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html), which requires all files to have two sets of metadata in separate keyspaces.
+
+Keyspaces can hold approximately 17 trillion key-value pairs and have the following requirements:
+
+* All keys within a keyspace must be unique.
+* The keyspace and key are required to create or access a {{site.udm.l}} entry.
+
+
+## Managing {{site.udm.t}} by Using the qq CLI
+This section explains how to create, retrieve, list, and delete {{site.udm.l}} by using the `qq` CLI.
+
+{{site.data.alerts.note}}
+<ul>
+  <li>All <code>qq</code> CLI commands default to using the <a href="#keyspace"><code>GENERIC</code> keyspace</a>. For the <code>S3</code> keyspace, use the <code>--s3</code> flag.</li>
+  <li>In the following examples, you can specify the file path by using the <code>--path</code> flag or the file ID by using the <code>--id</code> flag.</li>
+</ul>
+{{site.data.alerts.end}}
+
+### Prerequisites
+Managing {{site.udm.l}} requires the following privileges:
+
+* `READ_EA`: Read the {{site.udm.l}} from a file
+* `WRITE_EA`: Write to, or delete, the {{site.udm.l}} of a file
+
+### To Create a Generic {{site.udm.t}} Entry for a File by Using the qq CLI
+Use the `fs_set_user_metadata` command and specify the path to the file, the key, and the value. For example:
+
+```bash
+qq fs_set_user_metadata \
+  --path my-file \
+  --key my-key \
+  --value my-value
+```
+
+{{site.udm.hex}}
+
+For more information, see [`qq fs_set_user_metadata`](https://docs.qumulo.com/qq-cli-command-guide/fs/fs_set_user_metadata.html) in the {{site.guides.cli}}.
+
+### To Retrieve a Generic {{site.udm.t}} Entry for a File by Using the qq CLI
+Use `fs_get_user_metadata` command and specify the path to the file and the key. For example:
+
+```bash
+qq fs_get_user_metadata \
+  --path my-file \
+  --key my-key
+```
+
+* {{site.udm.hex}}
+* {{site.udm.snap}}
+
+For more information, see [`qq fs_get_user_metadata`](https://docs.qumulo.com/qq-cli-command-guide/fs/fs_get_user_metadata.html) in the {{site.guides.cli}}.
+
+### To List All Generic {{site.udm.t}} Entries for a File by Using the qq CLI
+Use the `fs_list_user_metadata` command and specify the path to the file. For example:
+
+```bash
+qq fs_list_user_metadata \
+  --path my-file
+```
+
+* {{site.udm.hex}}
+* {{site.udm.snap}}
+
+For more information, see [`qq fs_list_user_metadata`](https://docs.qumulo.com/qq-cli-command-guide/fs/fs_list_user_metadata.html) in the {{site.guides.cli}}.
+
+### To Delete a Generic {{site.udm.t}} Entry for a File by Using the qq CLI
+Use the `fs_delete_user_metadata` command and specify the path to the file and the key. For example:
+
+```bash
+  qq fs_delete_user_metadata \
+    --path my-file \
+    --key my-key
+```
+
+For more information, see [`qq fs_delete_user_metadata`](https://docs.qumulo.com/qq-cli-command-guide/fs/fs_delete_user_metadata.html) in the {{site.guides.cli}}.
+
+
+## Managing {{site.udm.t}} by Using the S3 API
+S3 categorizes metadata as:
+
+* **Metadata**
+
+  * Immutable metadata that remains for the life of the object.
+
+  * Qumulo Core maps metadata to the [`S3` keyspace](#keyspace).
+
+* **Tags**
+
+  * Mutable metadata that doesn't impact the object's entity tag.
+
+    {% include important.html content="Tag values that can't be encoded by using UTF-8 aren't visible to S3." %}
+
+  * Qumulo Core maps tags to the [`GENERIC` keyspace](#keyspace).
+
+In Qumulo Core 6.3.2 (and higher) the [Qumulo S3 API](../s3-api) supports {{site.udm.l}} fully. For more information about how to access metadata by using the S3 API, see the [Amazon Simple Storage Service
+API Reference](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html).

administrator-guide/metadata/index.md

@@ -0,0 +1,7 @@
+---
+title: "Metadata"
+layout: landing_page
+permalink: /administrator-guide/metadata/
+sidebar: administrator_guide_sidebar
+search: exclude
+---

administrator-guide/metadata/managing-user-defined-metadata.md

@@ -0,0 +1,9 @@
+---
+title: "Managing User-Defined Metadata in Qumulo Core"
+summary: "This section explains how to create, retrieve, list, and delete user-defined metadata in Qumulo Core by using the <code>qq</code> CLI."
+permalink: /administrator-guide/metadata/managing-user-defined-metadata.html
+sidebar: administrator_guide_sidebar
+keywords: user_defined_metadata, metadata, user_metadata, tag
+---
+
+{% include content-reuse/admin-guides/metadata/managing-user-defined-metadata.md %}

administrator-guide/metadata/user-metadata.md

@@ -0,0 +1,114 @@
+---
+title: "User Defined Metadata in Qumulo Core"
+summary: "This section documents User Defined Metadata Qumulo Core."
+permalink: /administrator-guide/user-metadata.html
+sidebar: administrator_guide_sidebar
+keywords: metadata
+---
+
+## What is User Defined Metadata
+
+User defined metadata is metadata that can be added to any file type stored by a Qumulo Core system,
+and is composed of user specified key-value pairs. The key is required to be a unicode string, and
+the value is a sequence of bytes. The total size of each key-value pair is required to be not more
+than 400KB, and each file has a key space that allows a large number of key-value pairs
+(approximately 17 trillion).
+
+### Keyspaces
+
+User defined metadata in Qumulo Core is divided into two separate keyspaces. These are referred to
+as `GENERIC` and `S3`. This distinction is mostly to support the S3 API, which is required to have
+two sets of metadata in separate keyspaces. It is required that, within a keyspace, all keys used in
+a key-value pair are unique. In order to access or create a user defiend metadata entry, the
+keyspace and key need to be specified. Keyspaces could be thought of as buckets of key-value pairs.
+
+## Working with User Defined Metadata
+
+User defined metadata is currently supported in the S3 and REST APIs in Qumulo Core, with qq
+commands. In order to read the user defined metadata of file, you will need to have the READ_EA
+privilege on the file. In order to write or delete the user defined metadata of a file, you will
+require the WRITE_EA privilege.
+
+### User Defined Metadata via qq
+
+All `qq` commands will default to using the `GENERIC` keyspace. Use the `--s3` option to access the
+S3 keyspace.
+
+As per all qq commands, either the path of the file can be specified with the `--path` option, or
+the file ID can be specified with the `--id` option.
+
+#### Setting User a Defined Metadata entry
+
+To set user defined metadata on a file, use the `fs_set_user_metadata` qq command, for example:
+
+  ```bash
+  $ qq fs_set_user_metadata \
+    --path my-file --key my-key --value my-value
+  ```
+
+This will add a key-value pair to the file `my-file`, with key `my-key` and value `my-value` in the
+`GENERIC` keyspace. Add the `--s3` flag to put the metadata into the `S3` keyspace. To specify a
+non-text value for the user defined metadata, use either the `--base64-value` or `--hex-value`
+options. For example:
+
+  ```bash
+  $ qq fs_set_user_metadata \
+    --path my-file --key my-key --hex-value A1B2C3D4 --s3
+  ```
+
+This will create a key-value pair with key `my-key` and hexdecimal value A1B2C3D4 in the `S3`
+keyspace of file `my-file`.
+
+#### Deleting a User Defined Metadata entry
+
+To delete a user defined metadata entry on a file, use the `fs_delete_user_metadata` qq command, for example:
+
+  ```bash
+  $ qq fs_delete_user_metadata \
+    --path my-file --key my-key
+  ```
+
+This will delete a key-value pair with key `my-key` from file `my-file` in the `GENERIC` keyspace,
+if such a key exists. Add the `--s3` flag to remove a value from the S3 keyspace.
+
+#### Listing User Defined Metadata
+
+To list all user defined metadata for a file, use the `fs_list_user_metadata` qq command. For
+example:
+
+  ```bash
+  $ qq fs_list_user_metadata --path my-file
+  ```
+
+This will list all user defined metadata in the `GENERIC` keyspace of the head version of file `my-file` in a
+tabular format. It is assumed that values can be utf-8 encoded. If you wish to display values in
+hex or base64 format, add the `--hex` or `--base64` flags. If you wish to display the user defined
+metadata in snapshot of the file, use the `--snapshot` flag with the id of the snapshot.
+
+#### Getting a User Defined Metadata entry
+
+To retrive a single user defined metadata value, use the `fs_get_user_metadata` qq command, for
+example:
+
+  ```bash
+  $ qq fs_get_user_metadata --path my-file --key my-key
+  ```
+
+This will return the value of the metadata entry of head version of file `my-file` with key `my-key`
+in the `GENERIC` keyspace. As per the `fs_list_user_metadata` command, the value will be displayed
+`utf-8` encoded if possible. If you wish to display this in hex or base64 format, add the `--hex` or
+`--base64` flags. If you wish to access the metadata in snapshot of the file, use the `--snapshot`
+flag with the id of the snapshot.
+
+### User Defined Metadata via S3 API
+
+S3 defines two types of user defined metadata, one type that is immutable for the life of the
+objecti (generally referred to as metadata by the S3 API), and another (called tags) that is mutable
+and will not impact the object etag. In Qumulo Core, tags are mapped to the `GENERIC` keyspace,
+whilst immutable metadata is mapped to the `S3` keyspace.
+
+{% include important.html content="Tags values are required to be utf-8 encodable." %}
+
+Consult the S3 API documentation, or the documentation of your S3 client for how to access metadata
+via the S3 API. The Qumulo Core S3 API has full support for user defined metadata from version 6.3.2
+and later.

azure-administrator-guide/metadata/index.md

@@ -0,0 +1,7 @@
+---
+title: "Metadata"
+layout: landing_page
+permalink: /azure-administrator-guide/metadata/
+sidebar: azure_guide_sidebar
+search: exclude
+---

azure-administrator-guide/metadata/managing-user-defined-metadata.md

@@ -0,0 +1,9 @@
+---
+title: "Managing User-Defined Metadata in Qumulo Core"
+summary: "This section explains how to create, retrieve, list, and delete user-defined metadata in Qumulo Core by using the <code>qq</code> CLI."
+permalink: /azure-administrator-guide/metadata/managing-user-defined-metadata.html
+sidebar: azure_guide_sidebar
+keywords: user_defined_metadata, metadata, user_metadata, tag
+---
+
+{% include content-reuse/admin-guides/metadata/managing-user-defined-metadata.md %}