[Task]: Improve Enrichment docs

[damccorm]

What needs to happen?

There are a few targeted fixes needed for the Enrichment docs:

• https://beam.apache.org/documentation/transforms/python/elementwise/enrichment/ should mention BigQuery
• https://beam.apache.org/documentation/transforms/python/elementwise/enrichment/ should describe how we do batching and how we do caching (with_redis_cache)
• https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb and https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/vertex_ai_feature_store_enrichment.ipynb should elaborate a little more on what a cross-join means in this context (maybe a picture would be nice?)
• In https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb the example handler with composite row key support link is dead

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[damccorm]

@claudevdm this would be good to pick up at some point when you have space (don't drop other things, just when this fits in nicely)

[Vishesh-Tripathi]

hello sir , i would like to work on this issue please assign me this one.

[liferoad]

Thanks! Please check https://beam.apache.org/contribute/:

Comment “.take-issue” on the issue you'd like to work on. This will cause the issue to be assigned to you.

[Vishesh-Tripathi]

.take-issue

[Vishesh-Tripathi]

hello sir i am updating the file by adding this section is it correct ?

BigQuery Support

The enrichment transform supports integration with BigQuery to dynamically enrich data using BigQuery datasets. By leveraging BigQuery as an external data source, users can execute efficient lookups for data enrichment directly in their Apache Beam pipelines.

To use BigQuery for enrichment:

• Configure your BigQuery table as the data source for the enrichment process.
• Ensure your pipeline has the appropriate credentials and permissions to access the BigQuery dataset.
• Specify the query to extract the data to be used for enrichment.

This integration is particularly beneficial for use cases that require augmenting real-time streaming data with information stored in BigQuery.

---

Batching

To optimize requests to external services, the enrichment transform uses batching. Instead of performing a lookup for each individual element, the transform groups multiple elements into a batch and performs a single lookup for the entire batch.

Advantages of Batching:

• Improved Throughput: Reduces the number of network calls.
• Lower Latency: Fewer round trips to the external service.
• Cost Optimization: Minimizes API call costs when working with paid external services.

Users can configure the batch size by specifying parameters in their pipeline setup. Adjusting the batch size can help fine-tune the balance between throughput and latency.

---

Caching with with_redis_cache

For frequently used enrichment data, caching can significantly improve performance by reducing repeated calls to the remote service. Apache Beam's with_redis_cache method allows you to integrate a Redis cache into the enrichment pipeline.

Benefits of Caching:

• Reduced Latency: Fetches enrichment data from the cache instead of making network calls.
• Improved Resilience: Minimizes the impact of network outages or service downtimes.
• Scalability: Handles large volumes of enrichment requests efficiently.

To enable caching:

1. Set up a Redis instance accessible by your pipeline.
2. Use the with_redis_cache method to configure the cache in your enrichment transform.
3. Specify the time-to-live (TTL) for cache entries to ensure data freshness.

Example:

from apache_beam.transforms.enrichment import with_redis_cache

# Enrichment pipeline with Redis cache
enriched_data = (input_data 
                 | 'Enrich with Cache' >> with_redis_cache(redis_config=redis_config, enrichment_transform=my_enrichment_transform))

[Vishesh-Tripathi]

and i am adding this section to explain Cross join . please tell me if there is any mistake or need of updating.

What is a Cross-Join?

A cross-join is a Cartesian product operation where each row from one table is combined with every row from another table. It is useful when we want to create all possible combinations of two datasets.

Example:

• Table A:

  A1	A2
  1	X
  2	Y

• Table B:

  B1	B2
  10	P
  20	Q

Result of Cross-Join:

A1	A2	B1	B2
1	X	10	P
1	X	20	Q
2	Y	10	P
2	Y	20	Q

Cross-joins can be computationally expensive for large datasets, so use them judiciously.

[Vishesh-Tripathi]

@damccorm sir where I can find correct URL to replace the broken one?

[damccorm]

I think that resource may have been deleted, but we could probably link to

beam/sdks/python/apache_beam/transforms/enrichment_handlers/bigtable.py

Line 41 in d50cc15

class BigTableEnrichmentHandler(EnrichmentSourceHandler[beam.Row, beam.Row]):

as an example of how to build a handler.

The other changes look reasonable to me at a high level, it is probably easier to just go ahead and open a PR when you have a chance though, that will make it a bit easier to see the difference and review it.

Thanks for doing this!

[mohamedawnallah]

Hi @Vishesh-Tripathi, are you currently working on this issue? I’d like to take it on otherwise. 🙏

cc: @damccorm, @liferoad

[Vishesh-Tripathi]

Ya I am working on the issue