Skip to content

Commit c4e15ee

Browse files
manotothmanototh
authored
Add make-series, mv-expand, parse-kv, parse-where, project-rename (#389)
1 parent cc14d84 commit c4e15ee

File tree

10 files changed

+675
-3
lines changed

10 files changed

+675
-3
lines changed

‎apl/apl-features.mdx‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -280,10 +280,15 @@ keywords: ['axiom documentation', 'documentation', 'axiom', 'APL', 'axiom proces
280280
| Tabular operator | [join](/apl/tabular-operators/join-operator) | Returns a dataset containing rows from two different tables based on conditions. |
281281
| Tabular operator | [limit](/apl/tabular-operators/limit-operator) | Returns the top N rows from the input dataset. |
282282
| Tabular operator | [lookup](/apl/tabular-operators/lookup-operator) | Returns a dataset where rows from one dataset are enriched with matching columns from a lookup table based on conditions. |
283+
| Tabular operator | [make-series](/apl/tabular-operators/make-series) | Returns a dataset where the specified field is aggregated into a time series. |
284+
| Tabular operator | [mv-expand](/apl/tabular-operators/mv-expand) | Returns a dataset where the specified field is expanded into multiple rows. |
283285
| Tabular operator | [order](/apl/tabular-operators/order-operator) | Returns the input dataset, sorted according to the specified fields and order. |
284286
| Tabular operator | [parse](/apl/tabular-operators/parse-operator) | Returns the input dataset with new fields added based on the specified parsing pattern. |
287+
| Tabular operator | [parse-kv](/apl/tabular-operators/parse-kv) | Returns a dataset where key-value pairs are extracted from a string field into individual columns. |
288+
| Tabular operator | [parse-where](/apl/tabular-operators/parse-where) | Returns a dataset where values from a string are extracted based on a pattern. |
285289
| Tabular operator | [project-away](/apl/tabular-operators/project-away-operator) | Returns the input dataset excluding the specified fields. |
286290
| Tabular operator | [project-keep](/apl/tabular-operators/project-keep-operator) | Returns a dataset with only the specified fields. |
291+
| Tabular operator | [project-rename](/apl/tabular-operators/project-rename) | Returns a dataset where the specified field is renamed according to the specified pattern. |
287292
| Tabular operator | [project-reorder](/apl/tabular-operators/project-reorder-operator) | Returns a table with the specified fields reordered as requested followed by any unspecified fields in their original order. |
288293
| Tabular operator | [project](/apl/tabular-operators/project-operator) | Returns a dataset containing only the specified fields. |
289294
| Tabular operator | [redact](/apl/tabular-operators/redact-operator) | Returns the input dataset with sensitive data replaced or hashed. |

‎apl/scalar-functions/conversion-functions/toarray.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ title: toarray
33
description: 'This page explains how to use the toarray function in APL.'
44
---
55

6-
Use the `toarray` function in APL to convert a dynamic-typed input—such as a bag, property bag, or JSON array—into a regular array. This is helpful when you want to process the elements individually with array functions like `array_length`, `array_index_of`, or `mv-expand`.
6+
Use the `toarray` function in APL to convert a dynamic-typed input—such as a bag, property bag, or JSON array—into a regular array. This is helpful when you want to process the elements individually with array functions like `array_length` or `array_index_of`.
77

88
You typically use `toarray` when working with semi-structured data, especially after parsing JSON from log fields or external sources. It lets you access and manipulate nested collections using standard array operations.
99

‎apl/tabular-operators/make-series.mdx‎

Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
---
2+
title: make-series
3+
description: 'This page explains how to use the make-series operator in APL.'
4+
---
5+
6+
## Introduction
7+
8+
The `make-series` operator transforms event data into array-based time series. Instead of producing one row per time bucket, `make-series` encodes the values and corresponding timestamps into arrays stored in table fields. This makes it possible to apply `series_*` functions for advanced manipulations such as moving averages, smoothing, anomaly detection, or other time-series computations.
9+
10+
You find this operator useful when you want to:
11+
12+
- Turn event data into array-encoded time series for further analysis.
13+
- Apply `series_*` functions (for example, `series_fir`, `series_stats`) to aggregated data.
14+
- Postprocess and then expand arrays back into rows with `mv-expand` for visualization or downstream queries.
15+
16+
Unlike `summarize`, which produces row-based aggregations, `make-series` is designed specifically for creating and manipulating array-based time series.
17+
18+
## For users of other query languages
19+
20+
If you come from other query languages, this section explains how to adjust your existing queries to achieve the same results in APL.
21+
22+
<AccordionGroup>
23+
<Accordion title="Splunk SPL users">
24+
25+
In Splunk SPL, the `timechart` command creates row-based time series, with one row per time bucket. In APL, the `make-series` operator instead encodes the series into arrays, which you can later manipulate or expand. This is a key difference from SPL’s row-based approach.
26+
27+
<CodeGroup>
28+
```sql Splunk example
29+
index=sample-http-logs
30+
| timechart span=1m avg(req_duration_ms)
31+
````
32+
33+
```kusto APL equivalent
34+
['sample-http-logs']
35+
| make-series avg(req_duration_ms) default=0 on _time from ago(1h) to now() step 1m
36+
```
37+
38+
</CodeGroup>
39+
40+
</Accordion>
41+
<Accordion title="ANSI SQL users">
42+
43+
In ANSI SQL, you typically use `GROUP BY` with a generated series or calendar table to create row-based time buckets. In APL, `make-series` creates arrays of values and timestamps in a single row. This lets you perform array-based computations on the time series before optionally expanding back into rows.
44+
45+
<CodeGroup>
46+
```sql SQL example
47+
SELECT
48+
time_bucket('1 minute', _time) AS minute,
49+
AVG(req_duration_ms) AS avg_duration
50+
FROM sample_http_logs
51+
WHERE _time > NOW() - interval '1 hour'
52+
GROUP BY minute
53+
ORDER BY minute
54+
```
55+
56+
```kusto APL equivalent
57+
['sample-http-logs']
58+
| make-series avg(req_duration_ms) default=0 on _time from ago(1h) to now() step 1m
59+
```
60+
61+
</CodeGroup>
62+
63+
</Accordion>
64+
</AccordionGroup>
65+
66+
## Usage
67+
68+
### Syntax
69+
70+
```kusto
71+
make-series [Aggregation [, ...]]
72+
[default = DefaultValue]
73+
on TimeField
74+
[in Range]
75+
step StepSize
76+
[by GroupingField [, ...]]
77+
```
78+
79+
### Parameters
80+
81+
| Parameter | Description |
82+
| ---------------- | --------------------------------------------------------------------------------------------------------------- |
83+
| `Aggregation` | One or more aggregation functions (for example, `avg()`, `count()`, `sum()`) applied to each time bin, producing arrays of values. |
84+
| `default` | A value to use when no records exist in a time bin. |
85+
| `TimeField` | The field containing timestamps used for binning. |
86+
| `Range` | An optional range expression specifying the start and end of the series (for example, `from ago(1h) to now()`). |
87+
| `StepSize` | The size of each time bin (for example, `1m`, `5m`, `1h`). |
88+
| `GroupingField` | Optional fields to split the series by, producing parallel arrays for each group. |
89+
90+
### Returns
91+
92+
The operator returns a table where each aggregation produces an array of values aligned with an array of time bins. Each row represents a group (if specified), with arrays that encode the entire time series.
93+
94+
## Use case examples
95+
96+
<Tabs>
97+
<Tab title="Log analysis">
98+
99+
You want to create an array-based time series of request counts, then compute a rolling average using a `series_*` function, and finally expand back into rows for visualization.
100+
101+
**Query**
102+
103+
```kusto
104+
['sample-http-logs']
105+
| make-series count() on _time from now()-24h to now() step 5m
106+
| extend moving_avg_count=series_fir(count_, dynamic([1, 1, 1, 1, 1]))
107+
| mv-expand moving_avg_count to typeof(long), count_ to typeof(long), time to typeof(datetime)
108+
| project-rename _time=time
109+
| summarize avg(moving_avg_count), avg(count_) by bin(_time, 5m)
110+
```
111+
112+
[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D%20%7C%20make-series%20count()%20on%20_time%20from%20now()-24h%20to%20now()%20step%205m%20%7C%20extend%20moving_avg_count%3Dseries_fir(count_%2C%20dynamic(%5B1%2C%201%2C%201%2C%201%2C%201%5D))%20%7C%20mv-expand%20moving_avg_count%20to%20typeof(long)%2C%20count_%20to%20typeof(long)%2C%20time%20to%20typeof(datetime)%20%7C%20project-rename%20_time%3Dtime%20%7C%20summarize%20avg(moving_avg_count)%2C%20avg(count_)%20by%20bin(_time%2C%205m)%22%2C%22queryOptions%22%3A%7B%22quickRange%22%3A%221d%22%7D%7D)
113+
114+
**Output**
115+
116+
| _time | count_ | moving_avg_count |
117+
| ------------------- | ------ | ---------------- |
118+
| 2025-09-29T10:00:00 | 120 | 118 |
119+
| 2025-09-29T10:05:00 | 130 | 122 |
120+
| 2025-09-29T10:10:00 | 110 | 121 |
121+
122+
The query turns request counts into arrays, applies a smoothing function, and then expands the arrays back into rows for analysis.
123+
124+
</Tab>
125+
<Tab title="OpenTelemetry traces">
126+
127+
You want to analyze span durations per service, storing them as arrays for later manipulation.
128+
129+
**Query**
130+
131+
```kusto
132+
['otel-demo-traces']
133+
| make-series avg(duration) on _time from ago(2h) to now() step 10m by ['service.name']
134+
```
135+
136+
[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'otel-demo-traces'%5D%20%7C%20make-series%20avg(duration)%20on%20_time%20from%20ago(2h)%20to%20now()%20step%2010m%20by%20%5B'service.name'%5D%22%7D)
137+
138+
**Output**
139+
140+
| service.name | avg_duration | time |
141+
| --------------- | ----------------------------- | ----------------------- |
142+
| frontend | [20ms, 18ms, 22ms, 19ms, ...] | [2025-09-29T08:00, ...] |
143+
| checkout | [35ms, 40ms, 33ms, 37ms, ...] | [2025-09-29T08:00, ...] |
144+
145+
The query produces array-encoded time series per service, which you can further process with `series_*` functions.
146+
147+
</Tab>
148+
<Tab title="Security logs">
149+
150+
You want to analyze the rate of HTTP 500 errors in your logs per minute.
151+
152+
**Query**
153+
154+
```kusto
155+
['sample-http-logs']
156+
| where status == '500'
157+
| make-series count() default=0 on _time from ago(30m) to now() step 1m
158+
```
159+
160+
[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D%20%7C%20where%20status%20%3D%3D%20'500'%20%7C%20make-series%20count()%20default%3D0%20on%20_time%20from%20ago(30m)%20to%20now()%20step%201m%22%7D)
161+
162+
**Output**
163+
164+
| count_ | _time |
165+
| ------------------------ | ---------------------- |
166+
| [1489, 1428, 1517, 1462, 1509, ...] | ["2025-09-30T09:08:14.921301725Z", "2025-09-30T09:09:14.921301725Z", ...] |
167+
168+
The query generates a time series of HTTP 500 error counts as an array-based time series for further analysis with `series_*` functions.
169+
170+
</Tab>
171+
</Tabs>
172+
173+
## List of related operators
174+
175+
- [extend](/apl/tabular-operators/extend-operator): Creates new calculated fields, often as preparation before `make-series`. Use `extend` when you want to preprocess data for time series analysis.
176+
- [mv-expand](/apl/tabular-operators/mv-expand): Expands arrays into multiple rows. Use `mv-expand` to work with the arrays returned by `make-series`.
177+
- [summarize](/apl/tabular-operators/summarize-operator): Aggregates rows into groups but does not generate continuous time bins. Use `summarize` when you want flexible grouping without forcing evenly spaced intervals.
178+
- [top](/apl/tabular-operators/top-operator): Returns the top rows by a specified expression, not time series. Use `top` when you want to focus on the most significant values instead of trends over time.

‎apl/tabular-operators/mv-expand.mdx‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
---
2+
title: mv-expand
3+
description: 'This page explains how to use the mv-expand operator in APL.'
4+
---
5+
6+
The `mv-expand` operator expands dynamic arrays and property bags into multiple rows. Each element of the array or each property of the bag becomes its own row, while other columns are duplicated.
7+
8+
You use `mv-expand` when you want to analyze or filter individual values inside arrays or objects. This is especially useful when working with logs that include lists of values, OpenTelemetry traces that contain arrays of spans, or security events that group multiple attributes into one field.
9+
10+
## For users of other query languages
11+
12+
If you come from other query languages, this section explains how to adjust your existing queries to achieve the same results in APL.
13+
14+
<AccordionGroup>
15+
<Accordion title="Splunk SPL users">
16+
17+
In Splunk SPL, the `mvexpand` command expands multi-value fields into separate events. The APL `mv-expand` operator works in a very similar way, splitting array values into individual rows. The main difference is that APL explicitly works with dynamic arrays or property bags, while Splunk handles multi-value fields implicitly.
18+
19+
<CodeGroup>
20+
```sql Splunk example
21+
... | mvexpand request_uri
22+
````
23+
24+
```kusto APL equivalent
25+
['sample-http-logs']
26+
| mv-expand uri
27+
```
28+
29+
</CodeGroup>
30+
31+
</Accordion>
32+
<Accordion title="ANSI SQL users">
33+
34+
In ANSI SQL, you use `CROSS JOIN UNNEST` or `CROSS APPLY` to flatten arrays into rows. In APL, `mv-expand` provides a simpler and more direct way to achieve the same result.
35+
36+
<CodeGroup>
37+
```sql SQL example
38+
SELECT id, value
39+
FROM logs
40+
CROSS JOIN UNNEST(request_uris) AS t(value)
41+
```
42+
43+
```kusto APL equivalent
44+
['sample-http-logs']
45+
| mv-expand uri
46+
```
47+
48+
</CodeGroup>
49+
50+
</Accordion>
51+
</AccordionGroup>
52+
53+
## Usage
54+
55+
### Syntax
56+
57+
```kusto
58+
mv-expand [kind=(bag|array)] [with_itemindex=IndexFieldName] FieldName [to typeof(Typename)] [limit Rowlimit]
59+
```
60+
61+
### Parameters
62+
63+
| Parameter | Description |
64+
| -------------------------------- | -------------------------------------------------------------------------------------- |
65+
| `kind` | Optional. Specifies whether the column is a bag (object) or an array. Defaults to `array`. |
66+
| `with_itemindex=IndexFieldName` | Optional. Outputs an additional column with the zero-based index of the expanded item. |
67+
| `FieldName` | Required. The name of the column that contains an array or object to expand. |
68+
| `to typeof(Typename)` | Optional. Converts each expanded element to the specified type. |
69+
| `limit Rowlimit` | Optional. Limits the number of expanded rows per record. |
70+
71+
### Returns
72+
73+
The operator returns a table where each element of the expanded array or each property of the expanded object is placed in its own row. Other columns are duplicated for each expanded row.
74+
75+
## Use case example
76+
77+
When analyzing logs, some values can be stored as arrays. You can use `mv-expand` to expand them into individual rows for easier filtering.
78+
79+
**Query**
80+
81+
```kusto
82+
['sample-http-logs']
83+
| limit 100
84+
| mv-expand territories
85+
| summarize count = count() by territory_name = tostring(territories)
86+
```
87+
88+
[Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D%20%7C%20limit%20100%20%7C%20mv-expand%20territories%20%7C%20summarize%20count%20%3D%20count()%20by%20territory_name%20%3D%20tostring(territories)%22%7D)
89+
90+
**Output**
91+
92+
| territory_name | count |
93+
| ---------------- | ------- |
94+
| United States | 67 |
95+
| India | 22 |
96+
| Japan | 12 |
97+
98+
This query expands the `territories` array into rows and counts the most frequent territories.
99+
100+
## List of related operators
101+
102+
- [project](/apl/tabular-operators/project-operator): Selects or computes columns. Use it when you want to reshape data, not expand arrays.
103+
- [summarize](/apl/tabular-operators/summarize-operator): Aggregates data across rows. Use it after expanding arrays to compute statistics.
104+
- [top](/apl/tabular-operators/top-operator): Returns the top N rows by expression. Use it after expansion to find the most frequent values.

0 commit comments

Comments
 (0)