Skip to content

Commit

Permalink
doc changes for ICT
Browse files Browse the repository at this point in the history
  • Loading branch information
dhruvarya-db committed Dec 16, 2024
1 parent fc81d12 commit 6ecd315
Show file tree
Hide file tree
Showing 4 changed files with 21 additions and 2 deletions.
7 changes: 7 additions & 0 deletions docs/source/delta-batch.md
Original file line number Diff line number Diff line change
Expand Up @@ -742,6 +742,13 @@ Each time a checkpoint is written, Delta automatically cleans up log entries old
.. note::
Due to log entry cleanup, instances can arise where you cannot time travel to a version that is less than the retention interval. <Delta> requires all consecutive log entries since the previous checkpoint to time travel to a particular version. For example, with a table initially consisting of log entries for versions [0, 19] and a checkpoint at verison 10, if the log entry for version 0 is cleaned up, then you cannot time travel to versions [1, 9]. Increasing the table property `delta.logRetentionDuration` can help avoid these situations.

### In-Commit Timestamps

Historically, Delta has relied on file modification timetamps to be the source of truth for when
the table was modified. This becomes problematic when tables are moved from one storage location to another since the file modification timestamps change in such scenarios. To ensure that the timestamps
used for time travel don't change in such scenarios and that timestamp-based time travel queries produce
consistent results, the [In-Commit Timestamps](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#in-commit-timestamps) table feature was introduced in Delta 3.3. This feature can be enabled by setting the table property `delta.enableInCommitTimestamps` to `true`. See the [Versioning](./versioning) section for more details around compatibility.

<a id="deltadataframewrites"></a>

## Write to a table
Expand Down
2 changes: 1 addition & 1 deletion docs/source/delta-drop-feature.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ You can drop the following Delta table features:
- `deletionVectors`. See [_](delta-deletion-vectors.md).
- `typeWidening-preview`. See [_](delta-type-widening.md). Type widening is available in preview in <Delta> 3.2.0 and above.
- `v2Checkpoint`. See [V2 Checkpoint Spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#v2-spec). Drop support for V2 Checkpoints is available in <Delta> 3.1.0 and above.

- `inCommitTimestamp`. See [In-Commit Timestamps Spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#in-commit-timestamps)
You cannot drop other [Delta table features](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#valid-feature-names-in-table-features).

## How are Delta table features dropped?
Expand Down
12 changes: 11 additions & 1 deletion docs/source/table-properties.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,16 @@ properties are set. Available Delta table properties include:
| |
| Default: `classic` |
+-------------------------------------------------------------------------------------------+

| `delta.enableInCommitTimestamps` |
| |
| `true` for enabling the InCommitTimestamps table feature. |
| |
| |
| See [_](/presto-integration.md#step-3-update-manifests). |
| |
| Data type: `Boolean` |
| |
| Default: `false` |
+-------------------------------------------------------------------------------------------+
.. <Delta> replace:: Delta Lake
.. <AS> replace:: Apache Spark
2 changes: 2 additions & 0 deletions docs/source/versioning.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ The following <Delta> features break forward compatibility. Features are enabled
Row Tracking, [Delta Lake 3.2.0](https://github.com/delta-io/delta/releases/tag/v3.2.0),[_](/delta-row-tracking.md)
Type widening (Preview),[Delta Lake 3.2.0](https://github.com/delta-io/delta/releases/tag/v3.2.0),[_](/delta-type-widening.md)
Identity columns, [Delta Lake 3.3.0](https://github.com/delta-io/delta/releases/tag/v3.3.0),[_](/delta-batch.md#use-identity-columns)
In-Commit Timestamps, [Delta Lake 3.3.0](https://github.com/delta-io/delta/releases/tag/v3.3.0),[_](/delta-batch.md#use-identity-columns)

<a id="table-protocol"></a>

Expand Down Expand Up @@ -113,6 +114,7 @@ The following table shows minimum protocol versions required for <Delta> feature
Vacuum Protocol Check,7,3,[Vacuum Protocol Check Spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#vacuum-protocol-check)
Row Tracking,7,3,[_](/delta-row-tracking.md)
Type widening (Preview),7,3,[_](/delta-type-widening.md)
In-Commit Timestamps,7,3,[In-Commit Timestamps Spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#in-commit-timestamps)

<a id="upgrade"></a>

Expand Down

0 comments on commit 6ecd315

Please sign in to comment.