A cool simple example of functional data engineering
It's the ideas of functional data engineering transferred to data engineering with the goal of getting easy to maintain & easy to reproducible data result.
For the rest I can only refer to Maxime Beauchemin:
- Functional Data Engineering
- A talk by Max on the topic
- Another talk by Max on FDE
- Finally this video for a deeper look into functional data engineering & Python.
The repository contains two examples of functional data engineering written in (very poor!) Python. The idea is to show you the simplest implementations of these practices without any distraction or fuss.
- Example 1 - making a Python load transform extract functional
- Example 2 - Encapsulating changing logic into data in Python
- Example 4 - Full functional mini data warehouse
- I've set up another great complete tutorial using Airflow and Python here
If you want to run this on your own machine, you can use batect.
Simply run ./batect dev and it will start the notebook server
for you, mounting everything into it.