Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ReadME: Table format updated for covered workflows #112

Merged
merged 1 commit into from
Jan 8, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 16 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
DataJourney demonstrates how organizations can effectively manage and utilize data by harnessing the power of open-source technologies. It's designed to help navigate the complex landscape of data tools, offering a structured approach to building **scalable**, and **reproducible** data workflows.

Built on open-source principles, the framework guides users through essential steps—from **identifying** goals and **selecting tools** to **testing** and **customising** workflows. With its flexible, modular design, DataJourney can be tailored to individual needs, making it an invaluable toolkit for data professionals.

Check notice on line 28 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L28

[Google.Passive] In general, use active voice instead of passive voice ('be tailored').
Raw output
{"message": "[Google.Passive] In general, use active voice instead of passive voice ('be tailored').", "location": {"path": "README.md", "range": {"start": {"line": 28, "column": 233}}}, "severity": "INFO"}
### 🧱 Design Philosophy (LEGO)
Built with additive, subtractive capabilities glued with open source.
Each layer has a certain strength of communication inbuilt
Expand All @@ -33,7 +33,7 @@
- PO (Base): Static home(s) to keep it together `(GitHub)`
- P1 (Tooling): Tooling, strings `(Powered by open source)`
- P2 (Maintenance + Monitoring): Env, automations `(Pixi + GHA)`
- P3 (Abstraction): Layer(s), CLI/task manager for users to interact with `(Pixi)`

Check failure on line 36 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L36

[Vale.Spelling] Did you really mean 'Env'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'Env'?", "location": {"path": "README.md", "range": {"start": {"line": 36, "column": 34}}}, "severity": "ERROR"}

Check failure on line 36 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L36

[Vale.Spelling] Did you really mean 'automations'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'automations'?", "location": {"path": "README.md", "range": {"start": {"line": 36, "column": 39}}}, "severity": "ERROR"}


![DJ Design](assets/design/dj_vision.png)
Expand All @@ -42,19 +42,22 @@
{✨= Experimental,
✅ = Implemented}

✅ `Python Packaging framework` design principles\
✅ `GitHub actions` configured\
✅ `Vale.sh` configured at PR level\
✅ `Pre-commit hooks` configured for code linting/formatting\
✨ `Hello world` LLM design example based on [LangChain](https://python.langchain.com/)\
✅ Environment management via [pixi](https://prefix.dev/)\
✅ Reading data from online sources using [intake](https://github.com/intake/intake)\
✅ Sample pipeline built using [Dagster](https://github.com/dagster-io/dagster)\
✅ Building Dashboard using [holoviews](https://holoviews.org/gallery/index.html) + [panel](https://panel.holoviz.org/reference/index.html)\
✅ Exploratory data analysis (EDA) using [mito](https://www.trymito.io/)\
✅ Web UI build on [Flask](https://flask.palletsprojects.com/en/3.0.x/) \
✅ Web UI re-done and expanded with [FastHTML](https://docs.fastht.ml/)\
✅ Leverage AI models to analyse data [GitHub AI models Beta](https://docs.github.com/en/github-models/prototyping-with-ai-models)
| Status | Workflow Description |
|--------|------------------------------------------------------------------------------------------------------------------|
| ✅ | `Python Packaging framework` design principles |
| ✅ | `GitHub actions` configured |
| ✅ | `Vale.sh` configured at PR level |
| ✅ | `Pre-commit hooks` configured for code linting/formatting |
| ✨ | `Hello world` LLM design example based on [LangChain](https://python.langchain.com/) |
| ✅ | Environment management via [pixi](https://prefix.dev/) |
| ✅ | Reading data from online sources using [intake](https://github.com/intake/intake) |

Check failure on line 53 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L53

[Vale.Spelling] Did you really mean 'pixi'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'pixi'?", "location": {"path": "README.md", "range": {"start": {"line": 53, "column": 39}}}, "severity": "ERROR"}
| ✅ | Sample pipeline built using [Dagster](https://github.com/dagster-io/dagster) |
| ✅ | Building Dashboard using [holoviews](https://holoviews.org/gallery/index.html) + [panel](https://panel.holoviz.org/reference/index.html) |

Check failure on line 55 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L55

[Vale.Spelling] Did you really mean 'Dagster'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'Dagster'?", "location": {"path": "README.md", "range": {"start": {"line": 55, "column": 40}}}, "severity": "ERROR"}
| ✅ | Exploratory data analysis (EDA) using [mito](https://www.trymito.io/) |

Check failure on line 56 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L56

[Vale.Spelling] Did you really mean 'holoviews'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'holoviews'?", "location": {"path": "README.md", "range": {"start": {"line": 56, "column": 37}}}, "severity": "ERROR"}
| ✅ | Web UI build on [Flask](https://flask.palletsprojects.com/en/3.0.x/) |

Check failure on line 57 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L57

[Vale.Spelling] Did you really mean 'mito'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'mito'?", "location": {"path": "README.md", "range": {"start": {"line": 57, "column": 50}}}, "severity": "ERROR"}
| ✅ | Web UI re-done and expanded with [FastHTML](https://docs.fastht.ml/) |
| ✅ | Leverage AI models to analyse data [GitHub AI models Beta](https://docs.github.com/en/github-models/prototyping-with-ai-models) |


### ☕️ Quickly getting started with DataJourney

Expand All @@ -63,8 +66,8 @@
- Added requirement to run the LLM workflows
- Switch directory `cd DataJourney`
- Download pixi : [prefix.dev](https://prefix.dev/)
- Activate env: `pixi shell`

Check failure on line 69 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L69

[Vale.Spelling] Did you really mean 'pixi'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'pixi'?", "location": {"path": "README.md", "range": {"start": {"line": 69, "column": 12}}}, "severity": "ERROR"}
- Install DJ framework locally `pixi run DJ_package`

Check failure on line 70 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L70

[Vale.Spelling] Did you really mean 'env'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'env'?", "location": {"path": "README.md", "range": {"start": {"line": 70, "column": 12}}}, "severity": "ERROR"}
- List all the tasks: `pixi task list`
- Execute a task from the list: `pixi run <TASK>`
- Execute a task with verbosity enabled: `pixi run -v <TASK>`
Expand All @@ -73,19 +76,19 @@

- GIT_TOKEN_CHECK
- DJ_package
- DJ_pre_commit

Check failure on line 79 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L79

[Vale.Spelling] Did you really mean 'DJ_package'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'DJ_package'?", "location": {"path": "README.md", "range": {"start": {"line": 79, "column": 3}}}, "severity": "ERROR"}
- DJ_dagster

Check failure on line 80 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L80

[Vale.Spelling] Did you really mean 'DJ_pre_commit'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'DJ_pre_commit'?", "location": {"path": "README.md", "range": {"start": {"line": 80, "column": 3}}}, "severity": "ERROR"}
- DJ_fasthtml_app

Check failure on line 81 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L81

[Vale.Spelling] Did you really mean 'DJ_dagster'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'DJ_dagster'?", "location": {"path": "README.md", "range": {"start": {"line": 81, "column": 3}}}, "severity": "ERROR"}
- DJ_flask_app

Check failure on line 82 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L82

[Vale.Spelling] Did you really mean 'DJ_fasthtml_app'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'DJ_fasthtml_app'?", "location": {"path": "README.md", "range": {"start": {"line": 82, "column": 3}}}, "severity": "ERROR"}
- DJ_mito_app

Check failure on line 83 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L83

[Vale.Spelling] Did you really mean 'DJ_flask_app'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'DJ_flask_app'?", "location": {"path": "README.md", "range": {"start": {"line": 83, "column": 3}}}, "severity": "ERROR"}
- DJ_panel_app

Check failure on line 84 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L84

[Vale.Spelling] Did you really mean 'DJ_mito_app'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'DJ_mito_app'?", "location": {"path": "README.md", "range": {"start": {"line": 84, "column": 3}}}, "severity": "ERROR"}
- DJ_llm_analysis

Check failure on line 85 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L85

[Vale.Spelling] Did you really mean 'DJ_panel_app'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'DJ_panel_app'?", "location": {"path": "README.md", "range": {"start": {"line": 85, "column": 3}}}, "severity": "ERROR"}

Check failure on line 86 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L86

[Vale.Spelling] Did you really mean 'DJ_llm_analysis'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'DJ_llm_analysis'?", "location": {"path": "README.md", "range": {"start": {"line": 86, "column": 3}}}, "severity": "ERROR"}

Check failure on line 87 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L87

[Vale.Spelling] Did you really mean 'DJ_hello_world_langchain'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'DJ_hello_world_langchain'?", "location": {"path": "README.md", "range": {"start": {"line": 87, "column": 3}}}, "severity": "ERROR"}
### 🔌 About pre-commit-hooks and activating
Just like the name suggests, pre-commit-hooks are designed to format the code based on PEP standards before committing. [More details](https://pre-commit.com/)

```shell

Check notice on line 91 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L91

[Google.Passive] In general, use active voice instead of passive voice ('are designed').
Raw output
{"message": "[Google.Passive] In general, use active voice instead of passive voice ('are designed').", "location": {"path": "README.md", "range": {"start": {"line": 91, "column": 47}}}, "severity": "INFO"}
pixi run DJ_pre_commit
```

Expand All @@ -97,12 +100,12 @@

### 🪼 Execute pre-configured Dagster pipeline

```shell

Check failure on line 103 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L103

[Vale.Spelling] Did you really mean 'Dagster'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'Dagster'?", "location": {"path": "README.md", "range": {"start": {"line": 103, "column": 30}}}, "severity": "ERROR"}
pixi run DJ_dagster
```
![Dagit UI output](assets/pipeline/dagster_ui.png)

### 🐙 Panel app

Check failure on line 108 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L108

[Vale.Spelling] Did you really mean 'Dagit'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'Dagit'?", "location": {"path": "README.md", "range": {"start": {"line": 108, "column": 3}}}, "severity": "ERROR"}
```shell
pixi run DJ_panel_app
```
Expand All @@ -110,11 +113,11 @@
*NOTE:*
The dashboard generated is exported into HTML format and saved as [stock_price_twilio_dashboard](analytics_framework%2Fdashboard%2Fstock_price_twilio_dashboard.html)

![Panel app output](assets/dashboard/panel_app_stock.png)

Check notice on line 116 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L116

[Google.Passive] In general, use active voice instead of passive voice ('is exported').
Raw output
{"message": "[Google.Passive] In general, use active voice instead of passive voice ('is exported').", "location": {"path": "README.md", "range": {"start": {"line": 116, "column": 25}}}, "severity": "INFO"}

Check failure on line 116 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L116

[Vale.Spelling] Did you really mean 'stock_price_twilio_dashboard'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'stock_price_twilio_dashboard'?", "location": {"path": "README.md", "range": {"start": {"line": 116, "column": 68}}}, "severity": "ERROR"}

### 🐵 Mito

To explore further visit [trymito.io](https://docs.trymito.io/)

Check failure on line 120 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L120

[Vale.Spelling] Did you really mean 'Mito'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'Mito'?", "location": {"path": "README.md", "range": {"start": {"line": 120, "column": 7}}}, "severity": "ERROR"}
```shell
pixi run DJ_mito_app
```
Expand All @@ -124,8 +127,8 @@
<div style="display: flex; justify-content: space-between;">
<img src="assets/pipeline/mito_graph.png" alt="mito_output" width="400"/>
<img src="assets/pipeline/mito_operations.png" alt="mito_output" width="400"/>
</div>

Check failure on line 130 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L130

[Vale.Spelling] Did you really mean 'mito_output'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'mito_output'?", "location": {"path": "README.md", "range": {"start": {"line": 130, "column": 52}}}, "severity": "ERROR"}

Check failure on line 131 in README.md

View workflow job for this annotation

GitHub Actions / vale

[vale] README.md#L131

[Vale.Spelling] Did you really mean 'mito_output'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'mito_output'?", "location": {"path": "README.md", "range": {"start": {"line": 131, "column": 57}}}, "severity": "ERROR"}
### 🦋 Display all data sources present via web UI

```shell
Expand Down
Loading