Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support snapshots configurations in stats collector, fix bugs where snapshot retention didn't support past DAY format, fix false positive edge case in snapshot event #291

Merged
merged 7 commits into from
Mar 4, 2025

Conversation

Will-Lo
Copy link
Collaborator

@Will-Lo Will-Lo commented Feb 26, 2025

Summary

Issue Briefly discuss the summary of the changes made in this
pull request in 2-3 lines.

We want to be able to emit events with user tables that have the history configuration for them so that monitoring systems can accurately detect when snapshot expiration fails for configured tables.
This PR also does some refactoring around parsing policy JSONs to be more robust and easier to manage as more policies are added.

This PR also fixes a bug where not all date granularities were supported in retention, since TimeUnit is maxed granularity at DAYS when technically the snapshot policy can be kept at higher granularities of MONTH and YEAR, although we currently still limit it to 3 DAYS.

We also add the secondOldestSnapshot field to the IcebergTableStats to handle the edge case where a table has very infrequent writes, this can cause snapshots to fall out of retention policy but not yet be cleaned up, if the secondOldestSnapshot is still recent then we should avoid being alerted.

Changes

  • Client-facing API Changes
  • Internal API Changes
  • Bug Fixes
  • New Features
  • Performance Improvements
  • Code Style
  • Refactoring
  • Documentation
  • Tests

For all the boxes checked, please include additional details of the changes made in this pull request.

Testing Done

  • Manually Tested on local docker setup. Please include commands ran, and their output.
  • Added new tests for the changes made.
  • Updated existing tests to reflect the changes made.
  • No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
  • Some other form of testing like staging or soak time in production. Please explain.

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.

Unit tests, tested snapshot version and ran tablestatscollector job in a cluster on Spark, did not see any errors.

Additional Information

  • Breaking Changes
  • Deprecations
  • Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

@Will-Lo Will-Lo changed the title [WIP] Support snapshots configurations in stats collector, fix bugs where s… Support snapshots configurations in stats collector, fix bugs where s… Feb 27, 2025
@Will-Lo Will-Lo changed the title Support snapshots configurations in stats collector, fix bugs where s… Support snapshots configurations in stats collector, fix bugs where snapshot retention didn't support past DAY format Feb 27, 2025
Copy link
Collaborator

@chenselena chenselena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very clean refactor Will, thanks
left a couple of questions regarding the old code

@Will-Lo Will-Lo changed the title Support snapshots configurations in stats collector, fix bugs where snapshot retention didn't support past DAY format Support snapshots configurations in stats collector, fix bugs where snapshot retention didn't support past DAY format, fix false positive edge case in snapshot event Mar 4, 2025
Copy link
Collaborator

@cbb330 cbb330 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

excellent, thanks for adding secondOldestSnapshot which will solve frequent false positives when monitoring the availability of Snapshot Expiration job.

@Will-Lo Will-Lo merged commit b423d63 into linkedin:main Mar 4, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants