diff --git a/content/en/reference_tables/_index.md b/content/en/reference_tables/_index.md index 7f6e731c93435..89216b2cc25bb 100644 --- a/content/en/reference_tables/_index.md +++ b/content/en/reference_tables/_index.md @@ -11,7 +11,14 @@ further_reading: - link: "/logs/explorer/advanced_search#filter-logs-based-on-reference-tables" tag: "Documentation" text: "Filter logs based on Reference Tables" -- link: "/cloud_cost_management/tag_pipelines/#map-multiple-tags" +- link: "/sheets/#lookup" + tag: "Documentation" + text: "Sheets lookup" +- link: "/service_management/events/pipelines_and_processors/lookup_processor/" + tag: "Documentation" + text: "Lookup processor for Events" +- link: "/cloud_cost_management/tag_pipelines/ +#map-multiple-tags" tag: "Documentation" text: "Use Reference Tables to add multiple tags to cost data" - link: 'https://www.datadoghq.com/blog/add-context-with-reference-tables/' @@ -28,23 +35,14 @@ Reference Tables allow you to combine custom metadata with information already i {{< img src="reference_tables/reference-table.png" alt="A reference table with data populated in the columns for org id, org name, parent org, account owner, and csm" style="width:100%;">}} -## Validation rules - -Reference Table names and column headers are validated using the following naming conventions and automatically updated or normalized, if necessary. - -| Rule | Normalization | -| ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | -| Names and headers cannot be duplicated. | Duplicated names are enumerated. For example, if `fileid` is used twice as a name, the first instance becomes `fileid1` and the second instance becomes `fileid2`. If a name or header is enumerated and it exceeds the 56 characters, it is rejected and needs to be renamed. | -| Names and headers cannot contain uppercase letters. | Names with uppercase letters are converted to lowercase. This conversion may result in duplicate names, which are then enumerated. For example, `Fileid` and `FileID` both become `fileid` and are enumerated to `fileid1` and `fileid2` respectively. | -| Names and headers cannot contain spaces. | Spaces other than leading and trailing spaces are replaced with underscore `_` characters. Leading and trailing spaces are removed. For example, `customer names` is replaced with `customer_names`. | -| Names and headers must start with a lowercase letter. | Uppercase characters are converted to lowercase. Non-letter leading characters are removed. For example, `23Two_three` becomes `two_three`. | -| Names and headers support only lowercase letters, numbers, and the `_` character. | Unsupported characters are replaced with the underscore `_` character, unless it breaks one of the rules above. In that case, the unsupported characters are normalized by the respective rule. | -| Names and headers must be 56 characters or less. | No normalization is done. Names and headers that have more than 56 characters are rejected and need to be renamed. | - ## Create a Reference Table +Datadog supports the following data sources, including integrations and manual CSV upload: + {{< tabs >}} -{{% tab "Manual upload" %}} +{{% tab "Cloud storage" %}} + +{{% collapse-content title="Manual upload" level="h4" expanded=true %}} Click **New Reference Table +**, then upload a CSV file, name the appropriate columns, and define the primary key for lookups. @@ -52,9 +50,9 @@ Click **New Reference Table +**, then upload a CSV file, name the appropriate co **Note**: The manual CSV upload method supports files up to 4MB. -{{% /tab %}} +{{% /collapse-content %}} -{{% tab "Amazon S3" %}} +{{% collapse-content title="Amazon S3" level="h4" id="amazon-s3" %}} Reference Tables can automatically pull a CSV file from an Amazon S3 bucket to keep your data up to date. The integration looks for changes to the CSV file in S3, and when the file is updated it replaces the Reference Table with the new data. This also enables API updating with the S3 API once the initial Reference Table is configured. **Note**: Reference Tables are not replaced if the content of the CSV file is unchanged. @@ -83,7 +81,7 @@ To update Reference Tables from S3, Datadog uses the IAM role in your AWS accoun "Version": "2012-10-17" } ``` -### Define the table +#### Define the table Click **New Reference Table +**, then add a name, select Amazon S3, fill out all fields, click import, and define the primary key for lookups. @@ -93,9 +91,9 @@ Click **New Reference Table +**, then add a name, select Amazon S3, fill out all [1]: https://app.datadoghq.com/account/settings#integrations/amazon-web-services [2]: https://docs.datadoghq.com/integrations/amazon_web_services/?tab=automaticcloudformation#installation -{{% /tab %}} -{{% tab "Azure storage" %}} +{{% /collapse-content %}} +{{% collapse-content title="Azure storage" level="h4" id="azure-storage" %}} 1. If you haven't already, set up the [Azure integration][1] within the subscription that holds the storage account from which you want to import your Reference Table. This involves [creating an app registration that Datadog can][2] integrate with. 2. In the Azure Portal, select the storage account that stores your Reference Table files. @@ -118,9 +116,10 @@ For more information, see the [Azure integration documentation][4]. [3]: https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-reader [4]: /integrations/azure/ -{{% /tab %}} +{{% /collapse-content %}} +{{% collapse-content title="Google Cloud storage" level="h4" id="google-cloud-storage" %}} -{{% tab "Google Cloud storage" %}} +### Google Cloud storage {{% site-region region="gov" %}}
Reference Tables are not available for your selected Datadog site ({{< region-param key="dd_site_name" >}})
@@ -136,7 +135,6 @@ For more information, see the [Azure integration documentation][4]. 1. In the window that appears, under the "New principals" field, enter the service account email that you created and added to the GCP tile in Step 1. Under "Assign roles", select the **Storage Object Viewer** role. Click **Save**. - {{< img src="reference_tables/grant_access.png" alt="Google Cloud console showing the configuration to grant access" style="width:100%;" >}} After reviewing and assigning the role, you can import into Reference Tables from Google Cloud. It may take a few minutes for your configuration to update in Datadog. @@ -148,11 +146,31 @@ After reviewing and assigning the role, you can import into Reference Tables fro [1]: /integrations/google_cloud_platform/#setup [2]: /integrations/google_cloud_platform/#1-create-your-google-cloud-service-account +{{% /collapse-content %}} + +{{% /tab %}} +{{% tab "Integrations" %}} + +{{< partial name="reference_tables/ref-tables-saas-integrations.html" >}} + {{% /tab %}} {{< /tabs >}} This Reference Table can be used to add additional attributes to logs with the [Lookup Processor][1]. +## Validation rules + +Reference Table names and column headers are validated using the following naming conventions and automatically updated or normalized, if necessary. + +| Rule | Normalization | +| ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | +| Names and headers cannot be duplicated. | Duplicated names are enumerated. For example, if `fileid` is used twice as a name, the first instance becomes `fileid1` and the second instance becomes `fileid2`. If a name or header is enumerated and it exceeds the 56 characters, it is rejected and needs to be renamed. | +| Names and headers cannot contain uppercase letters. | Names with uppercase letters are converted to lowercase. This conversion may result in duplicate names, which are then enumerated. For example, `Fileid` and `FileID` both become `fileid` and are enumerated to `fileid1` and `fileid2` respectively. | +| Names and headers cannot contain spaces. | Spaces other than leading and trailing spaces are replaced with underscore `_` characters. Leading and trailing spaces are removed. For example, `customer names` is replaced with `customer_names`. | +| Names and headers must start with a lowercase letter. | Uppercase characters are converted to lowercase. Non-letter leading characters are removed. For example, `23Two_three` becomes `two_three`. | +| Names and headers support only lowercase letters, numbers, and the `_` character. | Unsupported characters are replaced with the underscore `_` character, unless it breaks one of the rules above. In that case, the unsupported characters are normalized by the respective rule. | +| Names and headers must be 56 characters or less. | No normalization is done. Names and headers that have more than 56 characters are rejected and need to be renamed. | + ## Modify a Reference Table To modify an existing Reference Table with new data, select a table and click **Update Config** on the top right corner. @@ -202,6 +220,14 @@ You can create monitors from the **Monitors** tab, or click on the Settings icon Reach out to [support][5] if you have a use case that exceeds these limits. +## Automatic update frequency + +Reference Tables can be updated automatically, depending on the data source: + +- **Cloud file storage** (Amazon S3, Azure Storage, Google Cloud Storage): Every 5 minutes +- **Integrations**: Every hour +- **CSV manual uploads**: Automatic updates are not supported + ## Permissions ### Role based access diff --git a/layouts/partials/reference_tables/ref-tables-cloud-object-storage.html b/layouts/partials/reference_tables/ref-tables-cloud-object-storage.html new file mode 100644 index 0000000000000..69a5c444c6c48 --- /dev/null +++ b/layouts/partials/reference_tables/ref-tables-cloud-object-storage.html @@ -0,0 +1,53 @@ +{{ $dot := . }} +
+
+
+
+ +
+ {{ partial "img.html" (dict "root" . "src" "integrations_logos/amazon-s3_large.svg" "class" + "img-fluid" "alt" "amazon s3" "width" "200") }} +
+ +
+
+ +
+ {{ partial "img.html" (dict "root" . "src" "integrations_logos/azure-storage_large.svg" "class" + "img-fluid" "alt" "azure" "width" "200") }} +
+ +
+
+ +
+ {{ partial "img.html" (dict "root" . "src" + "reference_tables/csv-uploads-placeholder.png" "class" "img-fluid" "alt" + "custom" "width" "200") }} +
+ +
+
+ +
+ {{ partial "img.html" (dict "root" . "src" "integrations_logos/google-cloud-storage_large.svg" "class" + "img-fluid" "alt" "google cloud" "width" "200") }} +
+ +
+
+
+
+
\ No newline at end of file diff --git a/layouts/partials/reference_tables/ref-tables-saas-integrations.html b/layouts/partials/reference_tables/ref-tables-saas-integrations.html new file mode 100644 index 0000000000000..009c3de487d26 --- /dev/null +++ b/layouts/partials/reference_tables/ref-tables-saas-integrations.html @@ -0,0 +1,52 @@ +{{ $dot := . }} +
+
+
+
+ +
+ {{ partial "img.html" (dict "root" . "src" "integrations_logos/databricks_large.svg" "class" + "img-fluid" "alt" "databricks" "width" "200") }} +
+ +
+
+ +
+ {{ partial "img.html" (dict "root" . "src" "integrations_logos/salesforce_large.svg" "class" + "img-fluid" "alt" "salesforce" "width" "200") }} +
+ +
+
+ +
+ {{ partial "img.html" (dict "root" . "src" "integrations_logos/servicenow_large.svg" "class" + "img-fluid" "alt" "servicenow" "width" "200") }} +
+ +
+
+ +
+ {{ partial "img.html" (dict "root" . "src" "integrations_logos/snowflake_large.svg" "class" + "img-fluid" "alt" "snowflake" "width" "200") }} +
+ +
+
+
+
+
\ No newline at end of file diff --git a/static/images/integrations_logos/amazon-s3_large.svg b/static/images/integrations_logos/amazon-s3_large.svg new file mode 100644 index 0000000000000..499509f6dc41f --- /dev/null +++ b/static/images/integrations_logos/amazon-s3_large.svg @@ -0,0 +1,5 @@ + + + + + diff --git a/static/images/integrations_logos/azure-storage_large.svg b/static/images/integrations_logos/azure-storage_large.svg new file mode 100644 index 0000000000000..96cefe7331321 --- /dev/null +++ b/static/images/integrations_logos/azure-storage_large.svg @@ -0,0 +1,16 @@ + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/static/images/integrations_logos/databricks_large.svg b/static/images/integrations_logos/databricks_large.svg new file mode 100644 index 0000000000000..ef9333e2b7b31 --- /dev/null +++ b/static/images/integrations_logos/databricks_large.svg @@ -0,0 +1,4 @@ + + + + diff --git a/static/images/integrations_logos/google-cloud-storage_large.svg b/static/images/integrations_logos/google-cloud-storage_large.svg new file mode 100644 index 0000000000000..ee94ac80d3b72 --- /dev/null +++ b/static/images/integrations_logos/google-cloud-storage_large.svg @@ -0,0 +1,17 @@ + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/static/images/integrations_logos/salesforce_large.svg b/static/images/integrations_logos/salesforce_large.svg new file mode 100644 index 0000000000000..d72cf7717c5e1 --- /dev/null +++ b/static/images/integrations_logos/salesforce_large.svg @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/static/images/integrations_logos/servicenow_large.svg b/static/images/integrations_logos/servicenow_large.svg new file mode 100644 index 0000000000000..b3d750e03d22c --- /dev/null +++ b/static/images/integrations_logos/servicenow_large.svg @@ -0,0 +1,13 @@ + + + + + + + + + + + + + \ No newline at end of file diff --git a/static/images/integrations_logos/snowflake_large.svg b/static/images/integrations_logos/snowflake_large.svg new file mode 100644 index 0000000000000..8339d2ab24dce --- /dev/null +++ b/static/images/integrations_logos/snowflake_large.svg @@ -0,0 +1,3 @@ + + + diff --git a/static/images/reference_tables/csv-uploads-placeholder.png b/static/images/reference_tables/csv-uploads-placeholder.png new file mode 100644 index 0000000000000..f122d4cd685fb Binary files /dev/null and b/static/images/reference_tables/csv-uploads-placeholder.png differ