diff --git a/dashboard-elements/create.mdx b/dashboard-elements/create.mdx index 4bf41990..f9f9e537 100644 --- a/dashboard-elements/create.mdx +++ b/dashboard-elements/create.mdx @@ -148,15 +148,6 @@ _Array Fields_ - `exists` - `not-exists` -#### Special fields - -Axiom creates the following two fields automatically for a new dataset: - -- `_time` is the timestamp of the event. If the data you ingest doesn’t have a `_time` field, Axiom assigns the time of the data ingest to the events. -- `_sysTime` is the time when you ingested the data. - -In most cases, you can use `_time` and `_sysTime` interchangeably. The difference between them can be useful if you experience clock skews on your event-producing systems. - ### Group by (segmentation) When visualizing data, it can be useful to segment data into specific groups to more clearly understand how the data behaves. diff --git a/query-data/explore.mdx b/query-data/explore.mdx index f70bcc96..a97bdda0 100644 --- a/query-data/explore.mdx +++ b/query-data/explore.mdx @@ -119,15 +119,6 @@ To select the time range, choose one of the following options: - To the right of the **Start date** and **End date** fields, click Select date and time **Select date and time** to select specific times. - Enter the specific time in the **Start date** and **End date** fields using natural language. For example, `1 hour ago` or `last Monday at 3pm`. For more information on the supported natural language inputs, see the [Sugar documentation](https://sugarjs.com/docs/#/DateParsing). Don’t use the `from` and `after` keywords in your input. -### Special fields - -Axiom creates the following two fields automatically for a new dataset: - -- `_time` is the timestamp of the event. If the data you ingest doesn’t have a `_time` field, Axiom assigns the time of the data ingest to the events. -- `_sysTime` is the time when you ingested the data. - -In most cases, you can use `_time` and `_sysTime` interchangeably. The difference between them can be useful if you experience clock skews on your event-producing systems. - ## Create query using APL APL is a data processing language that supports filtering, extending, and summarizing data. For more information, see [Introduction to APL](/apl/introduction). diff --git a/reference/datasets.mdx b/reference/datasets.mdx index bb9ccf3d..01d261e0 100644 --- a/reference/datasets.mdx +++ b/reference/datasets.mdx @@ -62,14 +62,9 @@ Don’t create multiple Axiom organizations to separate your data. For example, -## Special fields +## Limits on ingested data -Axiom creates the following two fields automatically for a new dataset: - -- `_time` is the timestamp of the event. If the data you ingest doesn’t have a `_time` field, Axiom assigns the time of the data ingest to the events. -- `_sysTime` is the time when you ingested the data. - -In most cases, you can use `_time` and `_sysTime` interchangeably. The difference between them can be useful if you experience clock skews on your event-producing systems. +For more information on limits and requirements imposed by Axiom, see [Limits](/reference/field-restrictions). ## Create dataset diff --git a/reference/field-restrictions.mdx b/reference/field-restrictions.mdx index 12c40681..17d6dd55 100644 --- a/reference/field-restrictions.mdx +++ b/reference/field-restrictions.mdx @@ -5,8 +5,6 @@ sidebarTitle: Limits keywords: ['axiom documentation', 'documentation', 'axiom', 'reference', 'settings', 'field restrictions', 'time stamp', 'time stamp field', 'limits', 'requirements', 'pricing', 'usage'] --- -import IngestDataLimits from "/snippets/ingest-data-limits.mdx" - Axiom applies certain limits and requirements to guarantee good service across the platform. Some of these limits depend on your pricing plan, and some of them are applied system-wide. This reference article explains all limits and requirements applied by Axiom. Limits are necessary to prevent potential issues that could arise from the ingestion of excessively large events or data structures that are too complex. Limits help maintain system performance, allow for effective data processing, and manage resources effectively. @@ -45,7 +43,11 @@ For more information on how to save on data loading, data retention, and queryin Axiom restricts the number of datasets and the number of fields in your datasets. The number of datasets and fields you can use is based on your pricing plan and explained in the table above. -If you ingest a new event that would exceed the allowed number of fields in a dataset, Axiom returns an error and rejects the event. To prevent this error, ensure that the number of fields in your events are within the allowed limits. To reduce the number of fields in a dataset, [trim the dataset](/reference/datasets#trim-dataset) and [vacuum its fields](/reference/datasets#vacuum-fields). +If you ingest a new event that would exceed the allowed number of fields in a dataset, Axiom returns an error and rejects the event. To prevent this error, ensure that the number of fields in your events are within the allowed limits. + +To reduce the number of fields in a dataset, use one of the following approaches: +- [Trim the dataset](/reference/datasets#trim-dataset) and [vacuum its fields](/reference/datasets#vacuum-fields). +- Use [map fields](/apl/data-types/map-fields). ## System-wide limits @@ -53,7 +55,45 @@ The following limits are applied to all accounts, irrespective of the pricing pl ### Limits on ingested data - +The table below summarizes the limits Axiom applies to each data ingest. These limits are independent of your pricing plan. + +| | Limit | +| ---------------------------- | --------- | +| Maximum field size | 1 MB | +| Maximum events in a batch | 10,000 | +| Maximum field name length | 200 bytes | + +If you try to ingest data that exceeds these limits, Axiom does the following: +- Replaces strings that are too long with ``. +- Replaces binary with ``. +- Truncates maps and slices that nest deeper than 100 levels and replaces them with `nil` at the cut-off level. +- Converts the following float values to `nil`: + - NaN + - +Infty + - -Infty + +### Special fields + +Axiom creates the following two fields automatically for a new dataset: + +- `_time` is the timestamp of the event. If the data you ingest doesn’t have a `_time` field, Axiom assigns the time of the data ingest to the events. If you ingest data using the [Ingest data](/restapi/endpoints/ingestIntoDataset) API endpoint, you can specify the timestamp field with the [timestamp-field](/restapi/endpoints/ingestIntoDataset#parameter-timestamp-field) parameter. +- `_sysTime` is the time when you ingested the data. + +In most cases, use `_time` to define the timestamp of events. In rare cases, if you experience clock skews on your event-producing systems, `_sysTime` can be useful. + +### Reserved field names + +Axiom reserves the following field names for internal use: + +- `_blockInfo` +- `_cursor` +- `_rowID` +- `_source` +- `_sysTime` + +Don’t ingest data that contains these fields names. If you try to ingest a field with a reserved name, Axiom renames the ingested field to `_user_FIELDNAME`. For example, if you try to ingest the field `_sysTime`, Axiom renames it to `_user_sysTime`. + +In general, avoid ingesting field names that start with `_`. ### Requirements for timestamp field diff --git a/restapi/api-limits.mdx b/restapi/api-limits.mdx index aa327b25..d24c4710 100644 --- a/restapi/api-limits.mdx +++ b/restapi/api-limits.mdx @@ -4,8 +4,6 @@ description: 'Learn how to limit the number of calls a user can make over a cert keywords: ['axiom documentation', 'documentation', 'axiom', 'axiom api', 'rest api', 'rest', 'rate limits', 'user', 'organization', 'query limits', 'ingest limits', 'requests', 'reset', 'scope', 'message'] --- -import IngestDataLimits from "/snippets/ingest-data-limits.mdx" - Axiom limits the number of calls a user (and their organization) can make over a certain period of time to ensure fair usage and to maintain the quality of service for everyone. Axiom systems closely monitor API usage and if a user exceeds any thresholds, Axiom @@ -62,4 +60,4 @@ which can efficiently manage the number of requests by aggregating data before s ## Limits on ingested data - \ No newline at end of file +For more information on limits and requirements imposed by Axiom, see [Limits](/reference/field-restrictions). diff --git a/send-data/methods.mdx b/send-data/methods.mdx index ca9cf2b2..1b3ca08f 100644 --- a/send-data/methods.mdx +++ b/send-data/methods.mdx @@ -7,7 +7,6 @@ keywords: ["send data", "ingest", "methods", "integrations", "opentelemetry", "v import ReplaceDomain from "/snippets/replace-domain.mdx" import ReplaceDatasetToken from "/snippets/replace-dataset-token.mdx" -import IngestDataLimits from "/snippets/ingest-data-limits.mdx" The easiest way to send your first event data to Axiom is with a direct HTTP request using a tool like `cURL`. @@ -106,8 +105,6 @@ The following examples show how to send data using OpenTelemetry from various la If you need an ingestion method that isn’t in the list above, [contact Axiom](https://www.axiom.co/contact). -### Limits on ingested data +## Limits on ingested data - - -For more information about limits and requirements, see [Limits](/reference/field-restrictions). +For more information on limits and requirements imposed by Axiom, see [Limits](/reference/field-restrictions). diff --git a/snippets/ingest-data-limits.mdx b/snippets/ingest-data-limits.mdx deleted file mode 100644 index d961af28..00000000 --- a/snippets/ingest-data-limits.mdx +++ /dev/null @@ -1,7 +0,0 @@ -The table below summarizes the limits Axiom applies to each data ingest. These limits are independent of your pricing plan. - -| | Limit | -| ---------------------------- | --------- | -| Maximum event size | 1 MB | -| Maximum events in a batch | 10,000 | -| Maximum field name length | 200 bytes | \ No newline at end of file