[BEAM-34076] Implement TTL-based caching for BigQuery table definitions

[Strikerrx01]

[BEAM-34076] Implement TTL-based caching for BigQuery table definitions

Description

This change implements a thread-safe caching mechanism for BigQuery table definitions
in the BigQueryWrapper class to address issue BEAM-34076. The implementation uses
a TTL (Time-To-Live) caching strategy to reduce BigQuery API calls while maintaining
data freshness.

Motivation

Previously, get_table() was called independently by each worker, leading to potential
BigQuery quota issues when multiple workers access the same table definitions. This
implementation significantly reduces API calls by reusing cached table definitions
when appropriate.

Modifications

• Added cachetools.TTLCache with configurable cache parameters
• Implemented thread-safety using threading.RLock
• Added methods to configure TTL and cache size limits
• Added cache invalidation mechanisms (global and selective clearing)
• Ensured proper cache behavior when changing TTL settings
• Preserved existing API behavior for backward compatibility

Tests

Added comprehensive unit tests that verify:

• Correct caching behavior for repeated table lookups
• Cache entry expiration based on TTL
• Thread-safety in concurrent environments
• Proper cache invalidation when configuration changes
• Selective cache clearing by project, dataset, or table

Documentation

Added method documentation for all new public methods and internal implementation
details to ensure maintainability and clarity.

Fixes #34076

Please Review

@apache/beam-maintainers

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[Strikerrx01]

R: @stankiewicz Thanks for the review. I've addressed your feedback:

1. Removed redundant field_type.upper() call since the type is already uppercase
2. Fixed temp dataset handling logic to preserve original behavior with temp_dataset_id or (BigQueryWrapper.TEMP_DATASET + self._temporary_table_suffix)
3. Added comprehensive test coverage for caching functionality in bigquery_tools_test.py

Could you please review the updated code in bigquery_tools.py and bigquery_tools_test.py? The new tests verify caching behavior, TTL validation, and thread safety.

Please let me know if you'd like me to explain anything further.

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

[derrickaw]

Hi @Strikerrx01, friendly ping to look at finishing this pr. Thanks!

[chamikaramj]

@Strikerrx01 any updates ?

[Strikerrx01]

@Strikerrx01 any updates ?

@chamikaramj I apologize for the lack of updates on this PR over the past week. I've been dealing with a medical emergency that prevented me from working on this. I'm now back and have just pushed an update with the formatting changes as requested. Thank you for your patience, and I'm ready to address any remaining feedback to get this PR merged.

[stankiewicz]

there are formatting issues -- https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-LintandFormattingChecks and run yapf, thanks!