Skip to content

Commit 80a95ef

Browse files
gaborgsomogyigaborgsomogyi
authored
[FLINK-37295][doc] Add state ttl migration compatibility to docs
1 parent 4b7d62b commit 80a95ef

File tree

4 files changed

+233
-5
lines changed

4 files changed

+233
-5
lines changed

‎docs/content.zh/docs/dev/datastream/fault-tolerance/state.md‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -278,8 +278,6 @@ Heap state backend 会额外存储一个包括用户状态以及时间戳的 Jav
278278

279279
- 暂时只支持基于 *processing time* 的 TTL。
280280

281-
- 尝试从 checkpoint/savepoint 进行恢复时，TTL 的状态（是否开启）必须和之前保持一致，否则会遇到 "StateMigrationException"。
282-
283281
- TTL 的配置并不会保存在 checkpoint/savepoint 中，仅对当前 Job 有效。
284282

285283
- 不建议checkpoint恢复前后将state TTL从短调长，这可能会产生潜在的数据错误。
@@ -451,6 +449,15 @@ RocksDB backend 的默认后台清理策略会每处理 1000 条数据进行一
451449
- 对已有的作业，这个清理方式可以在任何时候通过 `StateTtlConfig` 启用或禁用该特性，比如从 savepoint 重启后。
452450
- 定期压缩功能只在 TTL 启用时生效。
453451

452+
### TTL Migration Compatibility
453+
454+
Starting from Flink 2.2.0, Flink supports seamless migration between state with and without TTL enabled.
455+
456+
If you previously configured state without TTL and now want to enable it (or vice versa),
457+
this is now safe to do without triggering restore-time errors.
458+
459+
Read the full details here: [TTL / Non-TTL State Migration Compatibility]({{< ref "docs/dev/datastream/fault-tolerance/state_migration" >}})
460+
454461
## 算子状态 (Operator State)
455462

456463
*算子状态*（或者*非 keyed 状态*）是绑定到一个并行算子实例的状态。[Kafka Connector]({{< ref "docs/connectors/datastream/kafka" >}}) 是 Flink 中使用算子状态一个很具有启发性的例子。Kafka consumer 每个并行实例维护了 topic partitions 和偏移量的 map 作为它的算子状态。

‎docs/content.zh/docs/dev/datastream/fault-tolerance/state_migration.md‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
---
2+
title: "State TTL Migration Compatibility"
3+
weight: 1
4+
type: docs
5+
aliases:
6+
- /dev/stream/state/state_migration.html
7+
- /apis/streaming/state_migration.html
8+
---
9+
<!--
10+
Licensed to the Apache Software Foundation (ASF) under one
11+
or more contributor license agreements. See the NOTICE file
12+
distributed with this work for additional information
13+
regarding copyright ownership. The ASF licenses this file
14+
to you under the Apache License, Version 2.0 (the
15+
"License"); you may not use this file except in compliance
16+
with the License. You may obtain a copy of the License at
17+
18+
http://www.apache.org/licenses/LICENSE-2.0
19+
20+
Unless required by applicable law or agreed to in writing,
21+
software distributed under the License is distributed on an
22+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
23+
KIND, either express or implied. See the License for the
24+
specific language governing permissions and limitations
25+
under the License.
26+
-->
27+
28+
# State TTL Migration Compatibility
29+
30+
Starting with **Apache Flink 2.2.0**, the system supports seamless enabling or disabling of **State Time-to-Live (TTL)** for existing state.
31+
This enhancement removes prior limitations where a change in TTL configuration could cause a `StateMigrationException` during restore.
32+
33+
## Version Overview
34+
35+
| Flink Version | Change |
36+
| ------------- | ------------------------------------------------------------------------------- |
37+
| **2.0.0** | Introduced `TtlAwareSerializer` to support TTL/non-TTL serializer compatibility |
38+
| **2.1.0** | Added TTL migration support for **RocksDBKeyedStateBackend** |
39+
| **2.2.0** | Added TTL migration support for **HeapKeyedStateBackend** |
40+
41+
> Full TTL state migration support across all major state backends is available from Flink **2.2.0** onwards.
42+
43+
## Motivation
44+
45+
In earlier Flink versions, switching TTL on or off in a `StateDescriptor` resulted in incompatibility errors.
46+
This was because TTL-enabled state used a different serialization format than non-TTL state.
47+
48+
## Compatibility Behavior
49+
50+
With the changes introduced across versions 2.0.0 to 2.2.0:
51+
52+
* Flink can now restore state created **without TTL** using a descriptor **with TTL enabled**.
53+
* Flink can also restore state created **with TTL** using a descriptor **without TTL enabled**.
54+
55+
The serializers and state backends transparently handle the presence or absence of TTL metadata.
56+
57+
## Supported Migration Scenarios
58+
59+
| Migration Type | Available Since | Behavior |
60+
| -------------------------------------- | ----------------------------- | ------------------------------------------------------------------- |
61+
| Non-TTL state → TTL-enabled descriptor | 2.1.0 (RocksDB), 2.2.0 (Heap) | Previous state restored as non-expired. TTL applied on new updates/accesses. |
62+
| TTL state → Non-TTL descriptor | 2.1.0 (RocksDB), 2.2.0 (Heap) | TTL metadata is ignored. State becomes permanently visible. |
63+
64+
## Implementation Details
65+
66+
The compatibility is achieved through the following changes:
67+
68+
* **TtlAwareSerializer** (Flink 2.0.0): Wraps user serializers to support reading/writing TTL and non-TTL formats.
69+
* **Backend migration logic**:
70+
* RocksDBKeyedStateBackend (Flink 2.1.0)
71+
* HeapKeyedStateBackend (Flink 2.2.0)
72+
73+
These components check the metadata during restore and adapt accordingly.
74+
75+
## Limitations
76+
77+
* Changes to TTL **parameters** (e.g. expiration time, update behavior) are not always compatible. These may require serializer migration.
78+
* TTL is not applied retroactively. Existing entries restored from non-TTL state will only expire after their next access or update.
79+
* This compatibility assumes no other incompatible changes to the state serializer.
80+
81+
## Example
82+
83+
```java
84+
ValueStateDescriptor<String> descriptor = new ValueStateDescriptor<>("user-state", String.class);
85+
descriptor.enableTimeToLive(StateTtlConfig.newBuilder(Time.hours(1)).build());
86+
```
87+
88+
If this descriptor replaces an earlier one without TTL, the state will be restored successfully in Flink 2.2.0+ and TTL will be enforced going forward.
89+
90+
## Related Information
91+
92+
* [Working with State]({{< ref "docs/dev/datastream/fault-tolerance/state" >}})
93+
* [FLINK-32955 JIRA](https://issues.apache.org/jira/browse/FLINK-32955)
94+
95+
## FAQ
96+
97+
### Can I disable TTL after it was previously enabled?
98+
99+
Yes. Flink will restore the values and ignore any TTL expiration metadata.
100+
101+
### Is this supported in RocksDB and Heap backends?
102+
103+
Yes, RocksDB since 2.1.0, Heap since 2.2.0.
104+
105+
### Which Flink version fully supports TTL migration?
106+
107+
Flink 2.2.0 is the first version where all necessary support is available.
108+
109+
### Do I need to change anything in my savepoint?
110+
111+
No. The migration is handled internally by Flink, provided serializers are otherwise compatible.
112+
113+
{{< top >}}

‎docs/content/docs/dev/datastream/fault-tolerance/state.md‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -342,9 +342,6 @@ and a primitive long value in memory. The RocksDB state backend adds 8 bytes per
342342

343343
- Only TTLs in reference to *processing time* are currently supported.
344344

345-
- Trying to restore state, which was previously configured without TTL, using TTL enabled descriptor or vice versa
346-
will lead to compatibility failure and `StateMigrationException`.
347-
348345
- The TTL configuration is not part of check- or savepoints but rather a way of how Flink treats it in the currently running job.
349346

350347
- It is not recommended to restore checkpoint state with adjusting the ttl from a short value to a long value,
@@ -542,6 +539,15 @@ where at least the first element has expired to determine the offset of the next
542539
e.g. after restart from savepoint.
543540
- Periodic compaction could only work when TTL is enabled.
544541

542+
### TTL Migration Compatibility
543+
544+
Starting from Flink 2.2.0, Flink supports seamless migration between state with and without TTL enabled.
545+
546+
If you previously configured state without TTL and now want to enable it (or vice versa),
547+
this is now safe to do without triggering restore-time errors.
548+
549+
Read the full details here: [TTL / Non-TTL State Migration Compatibility]({{< ref "docs/dev/datastream/fault-tolerance/state_migration" >}})
550+
545551
## Operator State
546552

547553
*Operator State* (or *non-keyed state*) is state that is bound to one

‎docs/content/docs/dev/datastream/fault-tolerance/state_migration.md‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
---
2+
title: "State TTL Migration Compatibility"
3+
weight: 1
4+
type: docs
5+
aliases:
6+
- /dev/stream/state/state_migration.html
7+
- /apis/streaming/state_migration.html
8+
---
9+
<!--
10+
Licensed to the Apache Software Foundation (ASF) under one
11+
or more contributor license agreements. See the NOTICE file
12+
distributed with this work for additional information
13+
regarding copyright ownership. The ASF licenses this file
14+
to you under the Apache License, Version 2.0 (the
15+
"License"); you may not use this file except in compliance
16+
with the License. You may obtain a copy of the License at
17+
18+
http://www.apache.org/licenses/LICENSE-2.0
19+
20+
Unless required by applicable law or agreed to in writing,
21+
software distributed under the License is distributed on an
22+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
23+
KIND, either express or implied. See the License for the
24+
specific language governing permissions and limitations
25+
under the License.
26+
-->
27+
28+
# State TTL Migration Compatibility
29+
30+
Starting with **Apache Flink 2.2.0**, the system supports seamless enabling or disabling of **State Time-to-Live (TTL)** for existing state.
31+
This enhancement removes prior limitations where a change in TTL configuration could cause a `StateMigrationException` during restore.
32+
33+
## Version Overview
34+
35+
| Flink Version | Change |
36+
| ------------- | ------------------------------------------------------------------------------- |
37+
| **2.0.0** | Introduced `TtlAwareSerializer` to support TTL/non-TTL serializer compatibility |
38+
| **2.1.0** | Added TTL migration support for **RocksDBKeyedStateBackend** |
39+
| **2.2.0** | Added TTL migration support for **HeapKeyedStateBackend** |
40+
41+
> Full TTL state migration support across all major state backends is available from Flink **2.2.0** onwards.
42+
43+
## Motivation
44+
45+
In earlier Flink versions, switching TTL on or off in a `StateDescriptor` resulted in incompatibility errors.
46+
This was because TTL-enabled state used a different serialization format than non-TTL state.
47+
48+
## Compatibility Behavior
49+
50+
With the changes introduced across versions 2.0.0 to 2.2.0:
51+
52+
* Flink can now restore state created **without TTL** using a descriptor **with TTL enabled**.
53+
* Flink can also restore state created **with TTL** using a descriptor **without TTL enabled**.
54+
55+
The serializers and state backends transparently handle the presence or absence of TTL metadata.
56+
57+
## Supported Migration Scenarios
58+
59+
| Migration Type | Available Since | Behavior |
60+
| -------------------------------------- | ----------------------------- | ------------------------------------------------------------------- |
61+
| Non-TTL state → TTL-enabled descriptor | 2.1.0 (RocksDB), 2.2.0 (Heap) | Previous state restored as non-expired. TTL applied on new updates/accesses. |
62+
| TTL state → Non-TTL descriptor | 2.1.0 (RocksDB), 2.2.0 (Heap) | TTL metadata is ignored. State becomes permanently visible. |
63+
64+
## Limitations
65+
66+
* Changes to TTL **parameters** (e.g. expiration time, update behavior) are not always compatible. These may require serializer migration.
67+
* TTL is not applied retroactively. Existing entries restored from non-TTL state will only expire after their next access or update.
68+
* This compatibility assumes no other incompatible changes to the state serializer.
69+
70+
## Example
71+
72+
```java
73+
ValueStateDescriptor<String> descriptor = new ValueStateDescriptor<>("user-state", String.class);
74+
descriptor.enableTimeToLive(StateTtlConfig.newBuilder(Time.hours(1)).build());
75+
```
76+
77+
If this descriptor replaces an earlier one without TTL, the state will be restored successfully in Flink 2.2.0+ and TTL will be enforced going forward.
78+
79+
## Related Information
80+
81+
* [Working with State]({{< ref "docs/dev/datastream/fault-tolerance/state" >}})
82+
* [FLINK-32955 JIRA](https://issues.apache.org/jira/browse/FLINK-32955)
83+
84+
## FAQ
85+
86+
### Can I disable TTL after it was previously enabled?
87+
88+
Yes. Flink will restore the values and ignore any TTL expiration metadata.
89+
90+
### Is this supported in RocksDB and Heap backends?
91+
92+
Yes, RocksDB since 2.1.0, Heap since 2.2.0 but ForSt is not yet added.
93+
94+
### Which Flink version fully supports TTL migration?
95+
96+
Flink 2.2.0 is the first version where all necessary support is available.
97+
98+
### Do I need to change anything in my savepoint?
99+
100+
No. The migration is handled internally by Flink, provided serializers are otherwise compatible.
101+
102+
{{< top >}}

0 commit comments

Comments
 (0)