feat(ingest/s3): add table filtering

[eagle-25]

Description

1. s3_source trying ingest all folders in path_spec.include and this noisily triggers the warning messages.

path_spec.include: s3://bucket/path/{table}/{partition0}

s3 files
1. s3://bucket/shop_log/year=2025/month=01/day=01/0000.csv  # error
2. s3://bucket/visit_log/year=2025/month=01/day=02/0000.csv  # error
3. s3://bucket/user_log/2025/0000.csv

==== start s3 ingestion

--- noisy logs---

INFO - Processing folder: datahub-test/shop_log
INFO - Getting files from folder: s3://bucket/datahub-test/shop_log/year=2025/
WARNING - Unable to find any files in the folder s3://bucket/datahub-test/shop_log/year=2025/. Skipping...

INFO - Processing folder: datahub-test/visit_log
INFO - Getting files from folder: s3://bucket/datahub-test/visit_log/year=2025/
WARNING - Unable to find any files in the folder s3://bucket/datahub-test/visit_log/year=2025/. Skipping...

--- meaningful log ---

INFO - Processing folder: datahub-test/user_log
INFO - Getting files from folder: s3://bucket/datahub-test/user_log/2025/
INFO - Extracting table schema from file: s3://bucket/datahub-test/user_log/2025/ai_kpi.04_kst_v4_dag_runs.csv

2. It is not able to ingest some specific tables from a path. Though we have the ignore_patterns property, it is not a proper way because it will be wordy.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Files with missing lines	Coverage Δ
...hub/ingestion/source/data_lake_common/path_spec.py	85.03% <100.00%> (+0.75%)	⬆️
...ngestion/src/datahub/ingestion/source/s3/source.py	86.66% <100.00%> (+0.13%)	⬆️

... and 37 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f14c42d...22869f4. Read the comment docs.

[sgomezvillamor]

Other sources usually handle this filtering with the allow/deny patterns. So this should be:

    tables_pattern: AllowDenyPattern = Field(

You can find many examples in the code https://github.com/search?q=repo%3Adatahub-project%2Fdatahub+AllowDenyPattern+language%3APython&type=code&l=Python

[eagle-25]

Updated! Could you check again?

[sgomezvillamor]

comment on the new config parameter, so it is aligned with other sources

[eagle-25]

Changes

• remove allowed_tables and is_allowed_table() from PathSpec
• add tables_pattern: AllowDenyPattern to PathSpec
• use tables_pattern.allowed() to filter the tables
• add allow_tables and deny_tables to s3 integration test
• add unit tests for tables_pattern.allowed() to PathSpec().allowed() and PathSpec().allowed_dir()

[eagle-25]

The CI (Python Build / Python Wheels) seems broken, and merging this PR would fix it.