From 94839efb7a1249db731d347959f32fb44a9641dd Mon Sep 17 00:00:00 2001 From: Mano Toth Date: Wed, 20 Aug 2025 16:01:46 +0200 Subject: [PATCH 1/9] Add reserved field names --- dashboard-elements/create.mdx | 9 --------- query-data/explore.mdx | 9 --------- reference/datasets.mdx | 13 +++++++++++++ 3 files changed, 13 insertions(+), 18 deletions(-) diff --git a/dashboard-elements/create.mdx b/dashboard-elements/create.mdx index aaa40016..ae6e2169 100644 --- a/dashboard-elements/create.mdx +++ b/dashboard-elements/create.mdx @@ -139,15 +139,6 @@ _Array Fields_ - `exists` - `not-exists` -#### Special fields - -Axiom creates the following two fields automatically for a new dataset: - -- `_time` is the timestamp of the event. If the data you ingest doesn’t have a `_time` field, Axiom assigns the time of the data ingest to the events. -- `_sysTime` is the time when you ingested the data. - -In most cases, you can use `_time` and `_sysTime` interchangeably. The difference between them can be useful if you experience clock skews on your event-producing systems. - ### Group by (segmentation) When visualizing data, it can be useful to segment data into specific groups to more clearly understand how the data behaves. diff --git a/query-data/explore.mdx b/query-data/explore.mdx index 85d33c27..8665469f 100644 --- a/query-data/explore.mdx +++ b/query-data/explore.mdx @@ -118,15 +118,6 @@ To select the time range, choose one of the following options: - Use the **Quick range** items to quickly select popular time ranges. - Use the **Custom start/end date** fields to select specific times. -### Special fields - -Axiom creates the following two fields automatically for a new dataset: - -- `_time` is the timestamp of the event. If the data you ingest doesn’t have a `_time` field, Axiom assigns the time of the data ingest to the events. -- `_sysTime` is the time when you ingested the data. - -In most cases, you can use `_time` and `_sysTime` interchangeably. The difference between them can be useful if you experience clock skews on your event-producing systems. - ## Create query using APL APL is a data processing language that supports filtering, extending, and summarizing data. For more information, see [Introduction to APL](/apl/introduction). diff --git a/reference/datasets.mdx b/reference/datasets.mdx index 489e677f..84882e48 100644 --- a/reference/datasets.mdx +++ b/reference/datasets.mdx @@ -71,6 +71,19 @@ Axiom creates the following two fields automatically for a new dataset: In most cases, you can use `_time` and `_sysTime` interchangeably. The difference between them can be useful if you experience clock skews on your event-producing systems. +### Reserved field names + +Axiom reserves the following field names for internal use: + +- `_cursor` +- `_rowID` +- `_source` +- `_sysTime` + +Don’t ingest data that contains these fields names. If you try to ingest a field with a reserved name, Axiom renames the ingested field to `_user_FIELDNAME`. For example, if you try to ingest the field `_sysTime`, Axiom renames it to `_user_sysTime`. + +In general, avoid ingesting field names that start with `_`. + ## Create dataset To create a dataset using the Axiom app, follow these steps: From 3f10b8d4e699d98144fb2d4ab3160ebc2d8624d8 Mon Sep 17 00:00:00 2001 From: Mano Toth Date: Thu, 21 Aug 2025 11:15:15 +0200 Subject: [PATCH 2/9] Implement feedback --- reference/datasets.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/datasets.mdx b/reference/datasets.mdx index 84882e48..78a23e28 100644 --- a/reference/datasets.mdx +++ b/reference/datasets.mdx @@ -69,7 +69,7 @@ Axiom creates the following two fields automatically for a new dataset: - `_time` is the timestamp of the event. If the data you ingest doesn’t have a `_time` field, Axiom assigns the time of the data ingest to the events. - `_sysTime` is the time when you ingested the data. -In most cases, you can use `_time` and `_sysTime` interchangeably. The difference between them can be useful if you experience clock skews on your event-producing systems. +In most cases, use `_time` to define the timestamp of events. In rare cases, if you experience clock skews on your event-producing systems, `_sysTime` can be useful. ### Reserved field names From 68c64e49c052c724a5e352088e0019b906659787 Mon Sep 17 00:00:00 2001 From: Mano Toth Date: Thu, 21 Aug 2025 12:40:33 +0200 Subject: [PATCH 3/9] Update datasets.mdx --- reference/datasets.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/reference/datasets.mdx b/reference/datasets.mdx index 78a23e28..2fd40d2c 100644 --- a/reference/datasets.mdx +++ b/reference/datasets.mdx @@ -75,6 +75,7 @@ In most cases, use `_time` to define the timestamp of events. In rare cases, if Axiom reserves the following field names for internal use: +- `_blockInfo` - `_cursor` - `_rowID` - `_source` From 248c752d00b1f3bd3b5e146cbe2b5233efdffc2b Mon Sep 17 00:00:00 2001 From: Mano Toth Date: Wed, 3 Sep 2025 14:59:54 +0200 Subject: [PATCH 4/9] Unify information about ingest validation --- reference/datasets.mdx | 23 ++------------- reference/field-restrictions.mdx | 48 +++++++++++++++++++++++++++++--- restapi/api-limits.mdx | 4 +-- send-data/methods.mdx | 7 ++--- snippets/ingest-data-limits.mdx | 7 ----- 5 files changed, 49 insertions(+), 40 deletions(-) delete mode 100644 snippets/ingest-data-limits.mdx diff --git a/reference/datasets.mdx b/reference/datasets.mdx index 2fd40d2c..228309ab 100644 --- a/reference/datasets.mdx +++ b/reference/datasets.mdx @@ -62,28 +62,9 @@ Don’t create multiple Axiom organizations to separate your data. For example, -## Special fields +## Limits on ingested data -Axiom creates the following two fields automatically for a new dataset: - -- `_time` is the timestamp of the event. If the data you ingest doesn’t have a `_time` field, Axiom assigns the time of the data ingest to the events. -- `_sysTime` is the time when you ingested the data. - -In most cases, use `_time` to define the timestamp of events. In rare cases, if you experience clock skews on your event-producing systems, `_sysTime` can be useful. - -### Reserved field names - -Axiom reserves the following field names for internal use: - -- `_blockInfo` -- `_cursor` -- `_rowID` -- `_source` -- `_sysTime` - -Don’t ingest data that contains these fields names. If you try to ingest a field with a reserved name, Axiom renames the ingested field to `_user_FIELDNAME`. For example, if you try to ingest the field `_sysTime`, Axiom renames it to `_user_sysTime`. - -In general, avoid ingesting field names that start with `_`. +For more information on limits and requirements imposed by Axiom, see [Limits](/reference/field-restrictions). ## Create dataset diff --git a/reference/field-restrictions.mdx b/reference/field-restrictions.mdx index 12c40681..f0dc2fe0 100644 --- a/reference/field-restrictions.mdx +++ b/reference/field-restrictions.mdx @@ -5,8 +5,6 @@ sidebarTitle: Limits keywords: ['axiom documentation', 'documentation', 'axiom', 'reference', 'settings', 'field restrictions', 'time stamp', 'time stamp field', 'limits', 'requirements', 'pricing', 'usage'] --- -import IngestDataLimits from "/snippets/ingest-data-limits.mdx" - Axiom applies certain limits and requirements to guarantee good service across the platform. Some of these limits depend on your pricing plan, and some of them are applied system-wide. This reference article explains all limits and requirements applied by Axiom. Limits are necessary to prevent potential issues that could arise from the ingestion of excessively large events or data structures that are too complex. Limits help maintain system performance, allow for effective data processing, and manage resources effectively. @@ -45,7 +43,11 @@ For more information on how to save on data loading, data retention, and queryin Axiom restricts the number of datasets and the number of fields in your datasets. The number of datasets and fields you can use is based on your pricing plan and explained in the table above. -If you ingest a new event that would exceed the allowed number of fields in a dataset, Axiom returns an error and rejects the event. To prevent this error, ensure that the number of fields in your events are within the allowed limits. To reduce the number of fields in a dataset, [trim the dataset](/reference/datasets#trim-dataset) and [vacuum its fields](/reference/datasets#vacuum-fields). +If you ingest a new event that would exceed the allowed number of fields in a dataset, Axiom returns an error and rejects the event. To prevent this error, ensure that the number of fields in your events are within the allowed limits. + +To reduce the number of fields in a dataset, use one of the following approaches: +- [Trim the dataset](/reference/datasets#trim-dataset) and [vacuum its fields](/reference/datasets#vacuum-fields). +- Use [map fields](/apl/data-types/map-fields). ## System-wide limits @@ -53,7 +55,45 @@ The following limits are applied to all accounts, irrespective of the pricing pl ### Limits on ingested data - +The table below summarizes the limits Axiom applies to each data ingest. These limits are independent of your pricing plan. + +| | Limit | +| ---------------------------- | --------- | +| Maximum event size | 1 MB | +| Maximum events in a batch | 10,000 | +| Maximum field name length | 200 bytes | + +If you try to ingest data that exceeds these limits, Axiom does the following: +- Replaces strings that are too long with . +- Replaces binary data that is too long with . +- Truncates maps and slices that nest deeper than 100 levels and replaces them with `nil` at the cut-off level. +- Converts the following float values to `nil`: + - NaN + - +Infty + - -Infty + +### Special fields + +Axiom creates the following two fields automatically for a new dataset: + +- `_time` is the timestamp of the event. If the data you ingest doesn’t have a `_time` field, Axiom assigns the time of the data ingest to the events. +- `_sysTime` is the time when you ingested the data. + +In most cases, use `_time` to define the timestamp of events. In rare cases, if you experience clock skews on your event-producing systems, `_sysTime` can be useful. + +### Reserved field names + +Axiom reserves the following field names for internal use: + +- `_blockInfo` +- `_cursor` +- `_rowID` +- `_source` +- `_sysTime` + +Don’t ingest data that contains these fields names. If you try to ingest a field with a reserved name, Axiom renames the ingested field to `_user_FIELDNAME`. For example, if you try to ingest the field `_sysTime`, Axiom renames it to `_user_sysTime`. + +In general, avoid ingesting field names that start with `_`. ### Requirements for timestamp field diff --git a/restapi/api-limits.mdx b/restapi/api-limits.mdx index aa327b25..d24c4710 100644 --- a/restapi/api-limits.mdx +++ b/restapi/api-limits.mdx @@ -4,8 +4,6 @@ description: 'Learn how to limit the number of calls a user can make over a cert keywords: ['axiom documentation', 'documentation', 'axiom', 'axiom api', 'rest api', 'rest', 'rate limits', 'user', 'organization', 'query limits', 'ingest limits', 'requests', 'reset', 'scope', 'message'] --- -import IngestDataLimits from "/snippets/ingest-data-limits.mdx" - Axiom limits the number of calls a user (and their organization) can make over a certain period of time to ensure fair usage and to maintain the quality of service for everyone. Axiom systems closely monitor API usage and if a user exceeds any thresholds, Axiom @@ -62,4 +60,4 @@ which can efficiently manage the number of requests by aggregating data before s ## Limits on ingested data - \ No newline at end of file +For more information on limits and requirements imposed by Axiom, see [Limits](/reference/field-restrictions). diff --git a/send-data/methods.mdx b/send-data/methods.mdx index ca9cf2b2..1b3ca08f 100644 --- a/send-data/methods.mdx +++ b/send-data/methods.mdx @@ -7,7 +7,6 @@ keywords: ["send data", "ingest", "methods", "integrations", "opentelemetry", "v import ReplaceDomain from "/snippets/replace-domain.mdx" import ReplaceDatasetToken from "/snippets/replace-dataset-token.mdx" -import IngestDataLimits from "/snippets/ingest-data-limits.mdx" The easiest way to send your first event data to Axiom is with a direct HTTP request using a tool like `cURL`. @@ -106,8 +105,6 @@ The following examples show how to send data using OpenTelemetry from various la If you need an ingestion method that isn’t in the list above, [contact Axiom](https://www.axiom.co/contact). -### Limits on ingested data +## Limits on ingested data - - -For more information about limits and requirements, see [Limits](/reference/field-restrictions). +For more information on limits and requirements imposed by Axiom, see [Limits](/reference/field-restrictions). diff --git a/snippets/ingest-data-limits.mdx b/snippets/ingest-data-limits.mdx deleted file mode 100644 index d961af28..00000000 --- a/snippets/ingest-data-limits.mdx +++ /dev/null @@ -1,7 +0,0 @@ -The table below summarizes the limits Axiom applies to each data ingest. These limits are independent of your pricing plan. - -| | Limit | -| ---------------------------- | --------- | -| Maximum event size | 1 MB | -| Maximum events in a batch | 10,000 | -| Maximum field name length | 200 bytes | \ No newline at end of file From dcdeae5d83c9a893d5053f0b2c21b2e2e21888a1 Mon Sep 17 00:00:00 2001 From: Mano Toth Date: Wed, 3 Sep 2025 15:01:15 +0200 Subject: [PATCH 5/9] Update field-restrictions.mdx --- reference/field-restrictions.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/field-restrictions.mdx b/reference/field-restrictions.mdx index f0dc2fe0..3dcb34ed 100644 --- a/reference/field-restrictions.mdx +++ b/reference/field-restrictions.mdx @@ -65,7 +65,7 @@ The table below summarizes the limits Axiom applies to each data ingest. These l If you try to ingest data that exceeds these limits, Axiom does the following: - Replaces strings that are too long with . -- Replaces binary data that is too long with . +- Replaces binary data that’s too long with . - Truncates maps and slices that nest deeper than 100 levels and replaces them with `nil` at the cut-off level. - Converts the following float values to `nil`: - NaN From a67340e8ec903e78cfd16eddaf44ed823a839386 Mon Sep 17 00:00:00 2001 From: Mano Toth Date: Wed, 3 Sep 2025 15:01:56 +0200 Subject: [PATCH 6/9] Update field-restrictions.mdx --- reference/field-restrictions.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/field-restrictions.mdx b/reference/field-restrictions.mdx index 3dcb34ed..e0727a16 100644 --- a/reference/field-restrictions.mdx +++ b/reference/field-restrictions.mdx @@ -64,8 +64,8 @@ The table below summarizes the limits Axiom applies to each data ingest. These l | Maximum field name length | 200 bytes | If you try to ingest data that exceeds these limits, Axiom does the following: -- Replaces strings that are too long with . -- Replaces binary data that’s too long with . +- Replaces strings that are too long with ``. +- Replaces binary data that’s too long with ``. - Truncates maps and slices that nest deeper than 100 levels and replaces them with `nil` at the cut-off level. - Converts the following float values to `nil`: - NaN From 4b85cf029197fc2c2bac3a59c10c89e6c3a83ce0 Mon Sep 17 00:00:00 2001 From: Mano Toth <71388581+manototh@users.noreply.github.com> Date: Fri, 5 Sep 2025 08:53:14 +0200 Subject: [PATCH 7/9] Apply suggestions from code review Co-authored-by: Mikkel Kamstrup Erlandsen --- reference/field-restrictions.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/field-restrictions.mdx b/reference/field-restrictions.mdx index e0727a16..31af46f3 100644 --- a/reference/field-restrictions.mdx +++ b/reference/field-restrictions.mdx @@ -65,7 +65,7 @@ The table below summarizes the limits Axiom applies to each data ingest. These l If you try to ingest data that exceeds these limits, Axiom does the following: - Replaces strings that are too long with ``. -- Replaces binary data that’s too long with ``. +- Replaces binary with ``. - Truncates maps and slices that nest deeper than 100 levels and replaces them with `nil` at the cut-off level. - Converts the following float values to `nil`: - NaN From 216f64e3de9760c4f5b37c38e30ef87b2d4e69c2 Mon Sep 17 00:00:00 2001 From: Mano Toth Date: Fri, 5 Sep 2025 10:03:56 +0200 Subject: [PATCH 8/9] Implement review --- reference/field-restrictions.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/field-restrictions.mdx b/reference/field-restrictions.mdx index 31af46f3..ee872212 100644 --- a/reference/field-restrictions.mdx +++ b/reference/field-restrictions.mdx @@ -76,7 +76,7 @@ If you try to ingest data that exceeds these limits, Axiom does the following: Axiom creates the following two fields automatically for a new dataset: -- `_time` is the timestamp of the event. If the data you ingest doesn’t have a `_time` field, Axiom assigns the time of the data ingest to the events. +- `_time` is the timestamp of the event. If the data you ingest doesn’t have a `_time` field, Axiom assigns the time of the data ingest to the events. If you ingest data using the [Ingest data](/restapi/endpoints/ingestIntoDataset) API endpoint, you can specify the timestamp field with the [timestamp-field](/restapi/endpoints/ingestIntoDataset#parameter-timestamp-field) parameter. - `_sysTime` is the time when you ingested the data. In most cases, use `_time` to define the timestamp of events. In rare cases, if you experience clock skews on your event-producing systems, `_sysTime` can be useful. From 4661031081cfac8b30b6b012a568c057f39ff127 Mon Sep 17 00:00:00 2001 From: Mano Toth Date: Fri, 5 Sep 2025 10:05:00 +0200 Subject: [PATCH 9/9] Update field-restrictions.mdx --- reference/field-restrictions.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/field-restrictions.mdx b/reference/field-restrictions.mdx index ee872212..17d6dd55 100644 --- a/reference/field-restrictions.mdx +++ b/reference/field-restrictions.mdx @@ -59,7 +59,7 @@ The table below summarizes the limits Axiom applies to each data ingest. These l | | Limit | | ---------------------------- | --------- | -| Maximum event size | 1 MB | +| Maximum field size | 1 MB | | Maximum events in a batch | 10,000 | | Maximum field name length | 200 bytes |