From e636d65a954e11827345bde264b53ac021be7d9a Mon Sep 17 00:00:00 2001 From: Adrian Date: Fri, 22 Mar 2024 17:40:54 +0100 Subject: [PATCH 1/5] blog post reverse etl --- .../blog/2024-03-25-reverse_etl_dlt.md | 196 ++++++++++++++++++ 1 file changed, 196 insertions(+) create mode 100644 docs/website/blog/2024-03-25-reverse_etl_dlt.md diff --git a/docs/website/blog/2024-03-25-reverse_etl_dlt.md b/docs/website/blog/2024-03-25-reverse_etl_dlt.md new file mode 100644 index 0000000000..9c28a5c7ca --- /dev/null +++ b/docs/website/blog/2024-03-25-reverse_etl_dlt.md @@ -0,0 +1,196 @@ +--- +slug: reverse-etl-dlt +title: "dlt adds Reverse ETL - build a custom destination in minutes" +image: https://storage.googleapis.com/dlt-blog-images/reverse-etl.png +authors: + name: Adrian Brudaru + title: Open source Data Engineer + url: https://github.com/adrianbr + image_url: https://avatars.githubusercontent.com/u/5762770?v=4 +tags: [reverse etl, pythonic] +--- + +# Pythonic reverse ETL is here + +## Why python is the right approach for doing Reverse ETL + +Reverse ETL is generally about putting data into a business app. This data would often come from a SQL database used as a middle layer for data integrations and calculations. + +That’s fine - but nowadays most data people speak python, and the types of things we want to put into an operational app don’t always come from a DB, they often come from other business apps, or from things like a dataframe on which we did some scoring, etc. + +![reverse etl](https://storage.googleapis.com/dlt-blog-images/reverse-etl.png) + +### The full potential of Reverse ETL is in the flexibility of sources + +Sql databases are a good start, but in reality very often our data source is something else. More often than not, it’s a python analyst’s implementation of some scoring, or some business calculation. + +Other times, it’s a business app - for example, we might have a form that sends the response data to a webhook, from where it could end up in Salesforce, DWH and Slack as a notification. And of course, if this is done by a data person it will be done in Python. + +Such, it follows that if we want to cater to the data crowd, we need to be pythonic. + +## There’s synergy with ETL + +Reverse ETL is ultimately ETL. Data is extracted from a source, it’s transformer, and then loaded to a destination. The challenges are similar, the most notable difference being in the fact that pulling data from a strongly typed environment like a db and converting it to weakly typed JSON is MUCH easier than the other way around. So in fact you can argue that Reverse ETL is simpler than ETL. + +### Flavors of Reverse ETL + +Just like we have ETL and ELT, we also have flavors of Reverse ETL + +- Reverse ETL, or TEL: Transform the data to a specification, read it from db and send it to the app. An example could be. +- Tool Reverse ETL or ETL: Extract from db, map fields to destination in the tool, load to destination. +- Pythonic Freestyle Reverse ETL: You extract data from wherever you want and put it anywhere except storage/db. Transformations optional. : + +Examples of Python reverse ETL + +- Read data from Mongo, do anomaly detection, notify anomalies to slack. +- Read membership data from Stripe, calculate chance to churn, upload to CRM for account managers. +- Capture a form response with a webhook and send the information to CRM, DWH and slack. + +## Add python? - new skills unlocked! + +So why is it much better to do reverse ETL in python? + +- **Live Streaming and Flexibility**: Python's ability to handle live data streams and integrate with various APIs and services surpasses the capabilities of SQL-based data warehouses designed for batch processing. +- **End-to-End Workflow**: Employing Python from data extraction to operational integration facilitates a streamlined workflow, enabling data teams to maintain consistency and efficiency across the pipeline. +- **Customization and Scalability**: Python's versatility allows for tailored solutions that can scale with minimal overhead, reducing the reliance on additional tools and simplifying maintenance. +- **Collaboration and Governance**: By keeping the entire data workflow within Python, teams can ensure better governance, compliance, and collaboration, leveraging common tools and repositories. + +## **Example: Building a Custom Destination and a pipeline in under 1h.** + +Documentation used: +Building a destination: [docs](https://dlthub.com/devel/dlt-ecosystem/destinations/destination) +SQL source: [docs](https://dlthub.com/devel/dlt-ecosystem/verified-sources/sql_database) +In this example, you will see why it’s faster to build a custom destination than set up a separate tool. + +DLT allows you to define custom destination functions. You'll write a function that extracts the relevant data from your dataframe and formats it for the Notion API. + +This example assumes you have set up Google Sheets API access and obtained the necessary credentials to authenticate.T + +### **Step 1: Setting Up Google Sheets API (10min)** + +1. Enable the Google Sheets API in the Google Developers Console. +2. Download the credentials JSON file. +3. Share the target Google Sheet with the email address found in your credentials JSON file. + +### **Step 2: Define the Destination method in its own file `sheets_destination.py` (20min)** + +Install the required package for Google API client: + +```bash +pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib + +``` + +Here’s how to define a destination function to update a Google Sheet: + +```python +import dlt +from google.oauth2.service_account import Credentials +from googleapiclient.discovery import build + +@dlt.destination(name="update_google_sheet", batch_size=10) +def append_to_google_sheets(items, table_schema, sheets_id: str = dlt.config.value("SHEETS_ID"), credentials_json: dict = dlt.secrets.value("GOOGLE_SHEETS_CREDENTIALS"), range_name: str = 'Sheet1!A1'): + """ + Send data to a Google Sheet. + :param items: Batch of items to send. + :param table_schema: Schema of the table (unused in this example but required by dlt). + :param sheets_id: ID of the Google Sheet, retrieved from config. + :param credentials_json: Google Service Account credentials, retrieved from secrets. + :param range_name: The specific range within the Sheet where data should be appended. + """ + credentials = Credentials.from_service_account_info(credentials_json) + service = build('sheets', 'v4', credentials=credentials) + + values = [ + [item['id'], item['name'], item['status']] for item in items # Adjust based on your data structure + ] + body = {'values': values} + result = service.spreadsheets().values().append( + spreadsheetId=sheets_id, range=range_name, + valueInputOption='RAW', insertDataOption='INSERT_ROWS', body=body + ).execute() + print(f"{result.get('updates').get('updatedRows')} rows have been added to the sheet.") + + +``` + +### **Step 3: Configure secrets (5min)** + +For the custom destination, you can follow this example. Configure the source as instructed in the source [documentation](https://dlthub.com/devel/dlt-ecosystem/verified-sources/shopify). + + **`secrets.toml`** + +```toml +[google_sheets] +credentials_json = ''' +{ + "type": "service_account", + "project_id": "your_project_id", + "private_key_id": "your_private_key_id", + ... +} +''' +``` + +**`config.toml`** + +```toml +[SHEETS] +id = "your_google_sheet_id_here" + +``` + +### **Step 4: Running the pieline in `my_pipeline.py`(10min)** + +Now, assuming you have a source function **`sql_database()`** from the verified sources, you can set up and run your pipeline as follows: + +```python + +import dlt +from dlt.common.destination import Destination +from your_source_module import sql_database # Adjust the import based on your actual source module + +# Assuming your source and destination setup as previously defined + +# Initialize the dlt pipeline with a unique name +pipeline = dlt.pipeline( + name="my_google_sheets_pipeline", + destination=Destination.from_reference( + destination_callable="sheets_destination.append_to_google_sheets" + ) +) + +# Use the source function and specify the resource "people_report" +source = sql_database().with_resources("people_report") + +# Now, run the pipeline with the specified source +# The write_disposition parameter might be unused depending on your destination function's implementation +info = pipeline.run(source=source) + +``` + +In this setup, **`send_to_google_sheets`** acts as a custom destination within your dlt pipeline, pushing the fetched data to the specified Google Sheet. This method enables streamlined and secure data operations, fully utilizing Python's capabilities for Reverse ETL processes into Google Sheets. + +## What does dlt do for me here? + +Using dlt for reverse ETL instead of plain Python, especially with its **`@dlt.destination`** decorator, provides you with a structured framework that streamlines the process of integrating data into various destinations. Here’s how the dlt decorator specifically aids you compared to crafting everything from scratch in plain Python: + +### Faster time to Production grade pipelines. + +The **`@dlt.destination`** decorator significantly reduces the need for custom boilerplate code. It provides a structured approach to manage batch processing, error handling, and retries, which would otherwise require complex custom implementations in plain Python. This built-in functionality ensures reliability and resilience in your data pipelines. + +### **Focus on custom business logic and adding value** + +The flexibility of creating custom destinations with dlt shifts the focus from the possibilities to the necessities of your specific use case. This empowers you to concentrate on implementing the best solutions for your unique business requirements. + +### **Scalability and efficient resource use** + +dlt facilitates efficient handling of large data loads through chunking and batching, allowing for optimal use of computing resources. This means even small worker machines can stream data effectively into your chosen endpoint instead of wasting a large machine waiting for network. The library design supports easy scaling and adjustments. Making changes to batch sizes or configurations is straightforward, ensuring your data pipelines can grow and evolve with minimal effort. This approach not only makes maintenance simpler but ensures that once a solution is implemented, it's broadly applicable across your projects. + +### **In Conclusion** + +Reverse ETL is just a piece of the ETL puzzle. It could be done cleaner and better when done in python end to end. + +Tools will always appeal to the non technical folks. However, anyone with ability to do python pipelines can do Reverse ETL pipelines too, bringing typical benefits of code vs tool to a dev team - customisation, collaboration, best practices etc. + +So read more about ~~how to built a dlt destination~~ and consider giving it a try in your next reverse ETL pipeline. \ No newline at end of file From e3e9abc7b1e756721f708f0b4b1dd1637eb73cf1 Mon Sep 17 00:00:00 2001 From: Adrian Date: Fri, 22 Mar 2024 17:42:37 +0100 Subject: [PATCH 2/5] typo --- docs/website/blog/2024-03-25-reverse_etl_dlt.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/website/blog/2024-03-25-reverse_etl_dlt.md b/docs/website/blog/2024-03-25-reverse_etl_dlt.md index 9c28a5c7ca..e4819cf65e 100644 --- a/docs/website/blog/2024-03-25-reverse_etl_dlt.md +++ b/docs/website/blog/2024-03-25-reverse_etl_dlt.md @@ -38,7 +38,7 @@ Just like we have ETL and ELT, we also have flavors of Reverse ETL - Reverse ETL, or TEL: Transform the data to a specification, read it from db and send it to the app. An example could be. - Tool Reverse ETL or ETL: Extract from db, map fields to destination in the tool, load to destination. -- Pythonic Freestyle Reverse ETL: You extract data from wherever you want and put it anywhere except storage/db. Transformations optional. : +- Pythonic Freestyle Reverse ETL: You extract data from wherever you want and put it anywhere except storage/db. Transformations optional. Examples of Python reverse ETL From 804c268a23e4edbd7a9eb424b8a78e7c3b9bc284 Mon Sep 17 00:00:00 2001 From: Adrian Date: Fri, 22 Mar 2024 17:43:20 +0100 Subject: [PATCH 3/5] format --- docs/website/blog/2024-03-25-reverse_etl_dlt.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/website/blog/2024-03-25-reverse_etl_dlt.md b/docs/website/blog/2024-03-25-reverse_etl_dlt.md index e4819cf65e..089757fcab 100644 --- a/docs/website/blog/2024-03-25-reverse_etl_dlt.md +++ b/docs/website/blog/2024-03-25-reverse_etl_dlt.md @@ -193,4 +193,4 @@ Reverse ETL is just a piece of the ETL puzzle. It could be done cleaner and bett Tools will always appeal to the non technical folks. However, anyone with ability to do python pipelines can do Reverse ETL pipelines too, bringing typical benefits of code vs tool to a dev team - customisation, collaboration, best practices etc. -So read more about ~~how to built a dlt destination~~ and consider giving it a try in your next reverse ETL pipeline. \ No newline at end of file +So read more about [how to built a dlt destination](https://dlthub.com/devel/dlt-ecosystem/destinations/destination) and consider giving it a try in your next reverse ETL pipeline. \ No newline at end of file From a92ccef0614bb2392d9fbe483e5ee684d5447eef Mon Sep 17 00:00:00 2001 From: Zaeem Athar Date: Tue, 26 Mar 2024 10:35:07 +0100 Subject: [PATCH 4/5] Update 2024-03-25-reverse_etl_dlt.md Fixing Typos --- .../blog/2024-03-25-reverse_etl_dlt.md | 42 +++++++++---------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/docs/website/blog/2024-03-25-reverse_etl_dlt.md b/docs/website/blog/2024-03-25-reverse_etl_dlt.md index 089757fcab..68cea1cf27 100644 --- a/docs/website/blog/2024-03-25-reverse_etl_dlt.md +++ b/docs/website/blog/2024-03-25-reverse_etl_dlt.md @@ -12,43 +12,43 @@ tags: [reverse etl, pythonic] # Pythonic reverse ETL is here -## Why python is the right approach for doing Reverse ETL +## Why Python is the right approach for doing Reverse ETL -Reverse ETL is generally about putting data into a business app. This data would often come from a SQL database used as a middle layer for data integrations and calculations. +Reverse ETL is generally about putting data into a business application. This data would often come from a SQL database used as a middle layer for data integrations and calculations. -That’s fine - but nowadays most data people speak python, and the types of things we want to put into an operational app don’t always come from a DB, they often come from other business apps, or from things like a dataframe on which we did some scoring, etc. +That’s fine - but nowadays most data people speak Python, and the types of things we want to put into an operational application don’t always come from a DB, they often come from other business applications, or from things like a dataframe on which we did some scoring, etc. ![reverse etl](https://storage.googleapis.com/dlt-blog-images/reverse-etl.png) ### The full potential of Reverse ETL is in the flexibility of sources -Sql databases are a good start, but in reality very often our data source is something else. More often than not, it’s a python analyst’s implementation of some scoring, or some business calculation. +SQL databases are a good start, but in reality very often our data source is something else. More often than not, it’s a Python analyst’s implementation of some scoring or some business calculation. -Other times, it’s a business app - for example, we might have a form that sends the response data to a webhook, from where it could end up in Salesforce, DWH and Slack as a notification. And of course, if this is done by a data person it will be done in Python. +Other times, it’s a business application - for example, we might have a form that sends the response data to a webhook, from where it could end up in Salesforce, DWH, and Slack as a notification. And of course, if this is done by a data person it will be done in Python. -Such, it follows that if we want to cater to the data crowd, we need to be pythonic. +Such, it follows that if we want to cater to the data crowd, we need to be Pythonic. ## There’s synergy with ETL -Reverse ETL is ultimately ETL. Data is extracted from a source, it’s transformer, and then loaded to a destination. The challenges are similar, the most notable difference being in the fact that pulling data from a strongly typed environment like a db and converting it to weakly typed JSON is MUCH easier than the other way around. So in fact you can argue that Reverse ETL is simpler than ETL. +Reverse ETL is ultimately ETL. Data is extracted from a source, its transformed, and then loaded to a destination. The challenges are similar, the most notable difference being that pulling data from a strongly typed environment like a DB and converting it to weakly typed JSON is MUCH easier than the other way around. So in fact you can argue that Reverse ETL is simpler than ETL. ### Flavors of Reverse ETL Just like we have ETL and ELT, we also have flavors of Reverse ETL -- Reverse ETL, or TEL: Transform the data to a specification, read it from db and send it to the app. An example could be. -- Tool Reverse ETL or ETL: Extract from db, map fields to destination in the tool, load to destination. -- Pythonic Freestyle Reverse ETL: You extract data from wherever you want and put it anywhere except storage/db. Transformations optional. +- **Reverse ETL or TEL:** Transform the data to a specification, read it from DB, and send it to the application. +- **Tool Reverse ETL or ETL:** Extract from DB, map fields to destination in the tool, load to destination. +- **Pythonic Freestyle Reverse ETL:** You extract data from wherever you want and put it anywhere except storage/DB. Transformations are optional. Examples of Python reverse ETL -- Read data from Mongo, do anomaly detection, notify anomalies to slack. -- Read membership data from Stripe, calculate chance to churn, upload to CRM for account managers. -- Capture a form response with a webhook and send the information to CRM, DWH and slack. +- Read data from Mongo, do anomaly detection, and notify anomalies to Slack. +- Read membership data from Stripe, calculate the chance to churn, and upload to CRM for account managers. +- Capture a form response with a webhook and send the information to CRM, DWH, and Slack. ## Add python? - new skills unlocked! -So why is it much better to do reverse ETL in python? +So why is it much better to do reverse ETL in Python? - **Live Streaming and Flexibility**: Python's ability to handle live data streams and integrate with various APIs and services surpasses the capabilities of SQL-based data warehouses designed for batch processing. - **End-to-End Workflow**: Employing Python from data extraction to operational integration facilitates a streamlined workflow, enabling data teams to maintain consistency and efficiency across the pipeline. @@ -64,7 +64,7 @@ In this example, you will see why it’s faster to build a custom destination th DLT allows you to define custom destination functions. You'll write a function that extracts the relevant data from your dataframe and formats it for the Notion API. -This example assumes you have set up Google Sheets API access and obtained the necessary credentials to authenticate.T +This example assumes you have set up Google Sheets API access and obtained the necessary credentials to authenticate. ### **Step 1: Setting Up Google Sheets API (10min)** @@ -74,7 +74,7 @@ This example assumes you have set up Google Sheets API access and obtained the n ### **Step 2: Define the Destination method in its own file `sheets_destination.py` (20min)** -Install the required package for Google API client: +Install the required package for the Google API client: ```bash pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib @@ -140,7 +140,7 @@ id = "your_google_sheet_id_here" ``` -### **Step 4: Running the pieline in `my_pipeline.py`(10min)** +### **Step 4: Running the pipeline in `my_pipeline.py`(10min)** Now, assuming you have a source function **`sql_database()`** from the verified sources, you can set up and run your pipeline as follows: @@ -185,12 +185,12 @@ The flexibility of creating custom destinations with dlt shifts the focus from t ### **Scalability and efficient resource use** -dlt facilitates efficient handling of large data loads through chunking and batching, allowing for optimal use of computing resources. This means even small worker machines can stream data effectively into your chosen endpoint instead of wasting a large machine waiting for network. The library design supports easy scaling and adjustments. Making changes to batch sizes or configurations is straightforward, ensuring your data pipelines can grow and evolve with minimal effort. This approach not only makes maintenance simpler but ensures that once a solution is implemented, it's broadly applicable across your projects. +dlt facilitates efficient handling of large data loads through chunking and batching, allowing for optimal use of computing resources. This means even small worker machines can stream data effectively into your chosen endpoint instead of wasting a large machine waiting for the network. The library design supports easy scaling and adjustments. Making changes to batch sizes or configurations is straightforward, ensuring your data pipelines can grow and evolve with minimal effort. This approach not only makes maintenance simpler but ensures that once a solution is implemented, it's broadly applicable across your projects. ### **In Conclusion** -Reverse ETL is just a piece of the ETL puzzle. It could be done cleaner and better when done in python end to end. +Reverse ETL is just a piece of the ETL puzzle. It could be done cleaner and better when done in Python end to end. -Tools will always appeal to the non technical folks. However, anyone with ability to do python pipelines can do Reverse ETL pipelines too, bringing typical benefits of code vs tool to a dev team - customisation, collaboration, best practices etc. +Tools will always appeal to the non-technical folks. However, anyone with the ability to do Python pipelines can do Reverse ETL pipelines too, bringing typical benefits of code vs tool to a dev team - customisation, collaboration, best practices, etc. -So read more about [how to built a dlt destination](https://dlthub.com/devel/dlt-ecosystem/destinations/destination) and consider giving it a try in your next reverse ETL pipeline. \ No newline at end of file +So read more about [how to built a dlt destination](https://dlthub.com/devel/dlt-ecosystem/destinations/destination) and consider giving it a try in your next reverse ETL pipeline. From 79146fbc22812e089c413ab94a43832f09ac6a2c Mon Sep 17 00:00:00 2001 From: Zaeem Athar Date: Tue, 26 Mar 2024 11:01:20 +0100 Subject: [PATCH 5/5] Update 2024-03-25-reverse_etl_dlt.md Fixing Heading Structure --- .../blog/2024-03-25-reverse_etl_dlt.md | 26 +++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/website/blog/2024-03-25-reverse_etl_dlt.md b/docs/website/blog/2024-03-25-reverse_etl_dlt.md index 68cea1cf27..31714d7fa4 100644 --- a/docs/website/blog/2024-03-25-reverse_etl_dlt.md +++ b/docs/website/blog/2024-03-25-reverse_etl_dlt.md @@ -30,13 +30,13 @@ Such, it follows that if we want to cater to the data crowd, we need to be Pytho ## There’s synergy with ETL -Reverse ETL is ultimately ETL. Data is extracted from a source, its transformed, and then loaded to a destination. The challenges are similar, the most notable difference being that pulling data from a strongly typed environment like a DB and converting it to weakly typed JSON is MUCH easier than the other way around. So in fact you can argue that Reverse ETL is simpler than ETL. +Reverse ETL is ultimately ETL. Data is extracted from a source, its transformed, and then loaded to a destination. The challenges are similar, the most notable difference being that pulling data from a strongly typed environment like a DB and converting it to weakly typed JSON is MUCH easier than the other way around. You can argue that Reverse ETL is simpler than ETL. ### Flavors of Reverse ETL Just like we have ETL and ELT, we also have flavors of Reverse ETL -- **Reverse ETL or TEL:** Transform the data to a specification, read it from DB, and send it to the application. +- **Reverse ETL or TEL:** Transform the data to a specification, read it from DB, and send it to an application. - **Tool Reverse ETL or ETL:** Extract from DB, map fields to destination in the tool, load to destination. - **Pythonic Freestyle Reverse ETL:** You extract data from wherever you want and put it anywhere except storage/DB. Transformations are optional. @@ -55,7 +55,7 @@ So why is it much better to do reverse ETL in Python? - **Customization and Scalability**: Python's versatility allows for tailored solutions that can scale with minimal overhead, reducing the reliance on additional tools and simplifying maintenance. - **Collaboration and Governance**: By keeping the entire data workflow within Python, teams can ensure better governance, compliance, and collaboration, leveraging common tools and repositories. -## **Example: Building a Custom Destination and a pipeline in under 1h.** +## Example: Building a Custom Destination and a pipeline in under 1h Documentation used: Building a destination: [docs](https://dlthub.com/devel/dlt-ecosystem/destinations/destination) @@ -66,13 +66,13 @@ DLT allows you to define custom destination functions. You'll write a function t This example assumes you have set up Google Sheets API access and obtained the necessary credentials to authenticate. -### **Step 1: Setting Up Google Sheets API (10min)** +### Step 1: Setting Up Google Sheets API (10min) 1. Enable the Google Sheets API in the Google Developers Console. 2. Download the credentials JSON file. 3. Share the target Google Sheet with the email address found in your credentials JSON file. -### **Step 2: Define the Destination method in its own file `sheets_destination.py` (20min)** +### Step 2: Define the Destination method in its own file `sheets_destination.py` (20min) Install the required package for the Google API client: @@ -114,7 +114,7 @@ def append_to_google_sheets(items, table_schema, sheets_id: str = dlt.config.val ``` -### **Step 3: Configure secrets (5min)** +### Step 3: Configure secrets (5min) For the custom destination, you can follow this example. Configure the source as instructed in the source [documentation](https://dlthub.com/devel/dlt-ecosystem/verified-sources/shopify). @@ -140,7 +140,7 @@ id = "your_google_sheet_id_here" ``` -### **Step 4: Running the pipeline in `my_pipeline.py`(10min)** +### Step 4: Running the pipeline in `my_pipeline.py`(10min) Now, assuming you have a source function **`sql_database()`** from the verified sources, you can set up and run your pipeline as follows: @@ -173,21 +173,21 @@ In this setup, **`send_to_google_sheets`** acts as a custom destination within y ## What does dlt do for me here? -Using dlt for reverse ETL instead of plain Python, especially with its **`@dlt.destination`** decorator, provides you with a structured framework that streamlines the process of integrating data into various destinations. Here’s how the dlt decorator specifically aids you compared to crafting everything from scratch in plain Python: +Using dlt for reverse ETL instead of plain Python, especially with its **`@dlt.destination`** decorator, provides a structured framework that streamlines the process of data integrating into various destinations. Here’s how the dlt decorator specifically aids you compared to crafting everything from scratch in plain Python: -### Faster time to Production grade pipelines. +### Faster time to Production grade pipelines The **`@dlt.destination`** decorator significantly reduces the need for custom boilerplate code. It provides a structured approach to manage batch processing, error handling, and retries, which would otherwise require complex custom implementations in plain Python. This built-in functionality ensures reliability and resilience in your data pipelines. -### **Focus on custom business logic and adding value** +### Focus on custom business logic and adding value The flexibility of creating custom destinations with dlt shifts the focus from the possibilities to the necessities of your specific use case. This empowers you to concentrate on implementing the best solutions for your unique business requirements. -### **Scalability and efficient resource use** +### Scalability and efficient resource use -dlt facilitates efficient handling of large data loads through chunking and batching, allowing for optimal use of computing resources. This means even small worker machines can stream data effectively into your chosen endpoint instead of wasting a large machine waiting for the network. The library design supports easy scaling and adjustments. Making changes to batch sizes or configurations is straightforward, ensuring your data pipelines can grow and evolve with minimal effort. This approach not only makes maintenance simpler but ensures that once a solution is implemented, it's broadly applicable across your projects. +dlt facilitates efficient handling of large data loads through chunking and batching, allowing for optimal use of computing resources. This means even small worker machines can stream data effectively into your chosen endpoint instead of wasting a large machine waiting for the network. The library design supports easy scaling and adjustments. Making changes to batch sizes or configurations is straightforward, ensuring your data pipelines can grow and evolve with minimal effort. This approach simplifies maintenance and ensures that once a solution is implemented, it's broadly applicable across your projects. -### **In Conclusion** +### In Conclusion Reverse ETL is just a piece of the ETL puzzle. It could be done cleaner and better when done in Python end to end.