[DOC][3.3] Doc changes for InCommitTimestamps #3979
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dhruvarya-db
changed the title
docs/source/delta-batch.md
Outdated
| ### In-Commit Timestamps | ||
|
|
||
| #### Overview | ||
| <Delta> 3.3 introduced [In-Commit Timestamps](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#in-commit-timestamps) to provide a more reliable and consistent way to track table modifications. This feature addresses limitations of the traditional approach that relied on file modification timestamps, particularly in scenarios involving data migration or replication. |
There was a problem hiding this comment.
Do we add delta-spark versions in our documentation?
There was a problem hiding this comment.
Looks like it has been done before. From the same file:
You can selectively overwrite only the data that matches an arbitrary expression. This feature is available with DataFrames in <Delta> 1.1.0 and above and supported in SQL in <Delta> 2.4.0 and above.
docs/source/delta-batch.md
Outdated
| <Delta> 3.3 introduced [In-Commit Timestamps](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#in-commit-timestamps) to provide a more reliable and consistent way to track table modifications. This feature addresses limitations of the traditional approach that relied on file modification timestamps, particularly in scenarios involving data migration or replication. | ||
|
|
||
| #### Background | ||
| Previously, <Delta> used file modification timestamps as the source of truth for table modifications. This approach presented several challenges: |
There was a problem hiding this comment.
Suggested change
| Previously, <Delta> used file modification timestamps as the source of truth for table modifications. This approach presented several challenges: | |
| Without the In-Commit Timestamp feature, <Delta> uses file modification timestamps as the commit timestamp. The commit timestamps are needed for various usecases e.g. time-travel to a specific time in the past. This approach has various limitations: |
docs/source/delta-batch.md
Outdated
| #### Overview | ||
| <Delta> 3.3 introduced [In-Commit Timestamps](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#in-commit-timestamps) to provide a more reliable and consistent way to track table modifications. This feature addresses limitations of the traditional approach that relied on file modification timestamps, particularly in scenarios involving data migration or replication. | ||
|
|
||
| #### Background |
There was a problem hiding this comment.
This is documentation and so we don't want to explain what delta used to do before and why this feature was built.
Instead we want to tell the behavior of Delta with and without this feature to users.
- Section-1: Overview
- Section-2: This could be renamed to Feature Details i.e. we can merge Background and Feature Details section.
- Inside
Feature Details, We can talk about how Delta behaves when the feature is enabled. - Inside
Feature Details, Next we can talk about how Delta behaves when feature is disabled + its limitations. - Then Section-3 - we can talk about how to enable the feature.
docs/source/delta-batch.md
Outdated
| 3. Time Travel Reliability: These timestamp changes could affect the accuracy and consistency of time travel queries | ||
|
|
||
| #### Enabling the Feature | ||
| This feature can be enabled by setting the table property `delta.enableInCommitTimestamps` to `true`: |
There was a problem hiding this comment.
Suggested change
| This feature can be enabled by setting the table property `delta.enableInCommitTimestamps` to `true`: | |
| This is a Writer table feature and can be enabled by setting the table property `delta.enableInCommitTimestamps` to `true`: |
There was a problem hiding this comment.
also attach the link to Writer TableFetaure section (if any).
prakharjain09
approved these changes
Dec 19, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
Updates docs with details about InCommitTimestamps.
How was this patch tested?
N/A
Does this PR introduce any user-facing changes?
No