Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update template files to comply with linter jobs. #249

Merged
merged 3 commits into from
Nov 4, 2024
Merged

Update template files to comply with linter jobs. #249

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
755273b
Lint?
lrcouto Oct 30, 2024
a3e26fa
Revert "Lint again"
lrcouto Oct 31, 2024
e8dbc00
Replace all Set, Tuple and Dict uses by python builtin type hints
lrcouto Oct 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion astro-airflow-iris/{{ cookiecutter.repo_name }}/docs/source/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-


# {{ cookiecutter.python_package }} documentation build
Expand Down
2 changes: 1 addition & 1 deletion astro-airflow-iris/{{ cookiecutter.repo_name }}/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ readme = "README.md"
dynamic = ["dependencies", "version"]

[project.scripts]
{{ cookiecutter.repo_name }} = "{{ cookiecutter.python_package }}.__main__:main"
"{{ cookiecutter.repo_name }}" = "{{ cookiecutter.python_package }}.__main__:main"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do the quotes not change any of the behaviour? We don't have these in the base template either.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have the same question. Is this a correct fix, it looks like ruff is treating this as pyproject.toml but it is using jinja syntax that ruff will not understand.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does not change the behavior from what I'm seeing here. I've tested it by creating a new project with a starter and using this branch as the --checkout argument. It resolves the repo name correctly and kedro run works as well.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested it locally as well and it does seem to work fine. I specifically checked that the resulting pyproject.toml file looked the same as when using the released version.


[project.entry-points."kedro.hooks"]

Expand Down
3 changes: 1 addition & 2 deletions ...s/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipeline_registry.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
"""Project pipelines."""
from typing import Dict

from kedro.framework.project import find_pipelines
from kedro.pipeline import Pipeline


def register_pipelines() -> Dict[str, Pipeline]:
def register_pipelines() -> dict[str, Pipeline]:
"""Register the project's pipelines.
Returns:
Expand Down
4 changes: 2 additions & 2 deletions ...er.repo_name }}/src/{{ cookiecutter.python_package }}/pipelines/data_engineering/nodes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@
PLEASE DELETE THIS FILE ONCE YOU START WORKING ON YOUR OWN PROJECT!
"""

from typing import Any, Dict
from typing import Any

import pandas as pd


def split_data(data: pd.DataFrame, example_test_data_ratio: float) -> Dict[str, Any]:
def split_data(data: pd.DataFrame, example_test_data_ratio: float) -> dict[str, Any]:
"""Node for splitting the classical Iris data set into training and test
sets, each split into features and labels.
The split ratio parameter is taken from conf/project/parameters.yml.
Expand Down
4 changes: 2 additions & 2 deletions ...cutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipelines/data_science/nodes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@
Delete this when you start working on your own Kedro project.
"""
import logging
from typing import Any, Dict
from typing import Any

import numpy as np
import pandas as pd


def train_model(
train_x: pd.DataFrame, train_y: pd.DataFrame, parameters: Dict[str, Any]
train_x: pd.DataFrame, train_y: pd.DataFrame, parameters: dict[str, Any]
) -> np.ndarray:
"""Node for training a simple multi-class logistic regression model. The
number of training iterations as well as the learning rate are taken from
Expand Down
2 changes: 1 addition & 1 deletion databricks-iris/{{ cookiecutter.repo_name }}/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ readme = "README.md"
dynamic = ["dependencies", "version"]

[project.scripts]
{{ cookiecutter.repo_name }} = "{{ cookiecutter.python_package }}.__main__:main"
"{{ cookiecutter.repo_name }}" = "{{ cookiecutter.python_package }}.__main__:main"

[project.entry-points."kedro.hooks"]

Expand Down
3 changes: 1 addition & 2 deletions databricks-iris/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/nodes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,13 @@
"""

import logging
from typing import Dict, Tuple

import numpy as np
import pandas as pd
from pyspark.sql import DataFrame


def split_data(data: DataFrame, parameters: Dict) -> Tuple:
def split_data(data: DataFrame, parameters: dict) -> tuple:
"""Splits data into features and targets training and test sets.
Args:
Expand Down
3 changes: 1 addition & 2 deletions ...s/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipeline_registry.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
"""Project pipelines."""
from typing import Dict

from kedro.framework.project import find_pipelines
from kedro.pipeline import Pipeline


def register_pipelines() -> Dict[str, Pipeline]:
def register_pipelines() -> dict[str, Pipeline]:
"""Register the project's pipelines.
Returns:
Expand Down
4 changes: 2 additions & 2 deletions features/environment.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
import tempfile
import venv
from pathlib import Path
from typing import Any, Set
from typing import Any

_PATHS_TO_REMOVE: Set[Path] = set()
_PATHS_TO_REMOVE: set[Path] = set()



Expand Down
1 change: 0 additions & 1 deletion spaceflights-pandas-viz/{{ cookiecutter.repo_name }}/docs/source/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-


# {{ cookiecutter.python_package }} documentation build
Expand Down
4 changes: 2 additions & 2 deletions spaceflights-pandas-viz/{{ cookiecutter.repo_name }}/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ readme = "README.md"
dynamic = ["dependencies", "version"]

[project.scripts]
{{ cookiecutter.repo_name }} = "{{ cookiecutter.python_package }}.__main__:main"
"{{ cookiecutter.repo_name }}" = "{{ cookiecutter.python_package }}.__main__:main"

[project.entry-points."kedro.hooks"]

Expand Down Expand Up @@ -43,7 +43,7 @@ namespaces = false
package_name = "{{ cookiecutter.python_package }}"
project_name = "{{ cookiecutter.project_name }}"
kedro_init_version = "{{ cookiecutter.kedro_version }}"
tools = {{ cookiecutter.tools | default('') | string | replace('\"', '\\\"') }}
tools = "{{ cookiecutter.tools | default('') | string | replace('\"', '\\\"') }}"
example_pipeline = "{{ cookiecutter.example_pipeline }}"
source_dir = "src"

Expand Down
3 changes: 1 addition & 2 deletions ...z/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipeline_registry.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
"""Project pipelines."""
from typing import Dict

from kedro.framework.project import find_pipelines
from kedro.pipeline import Pipeline


def register_pipelines() -> Dict[str, Pipeline]:
def register_pipelines() -> dict[str, Pipeline]:
"""Register the project's pipelines.
Returns:
Expand Down
3 changes: 1 addition & 2 deletions ...ter.repo_name }}/src/{{ cookiecutter.python_package }}/pipelines/data_processing/nodes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
from typing import Dict, Tuple

import pandas as pd

Expand All @@ -19,7 +18,7 @@ def _parse_money(x: pd.Series) -> pd.Series:
return x


def preprocess_companies(companies: pd.DataFrame) -> Tuple[pd.DataFrame, Dict]:
def preprocess_companies(companies: pd.DataFrame) -> tuple[pd.DataFrame, dict]:
"""Preprocesses the data for companies.

Args:
Expand Down
5 changes: 2 additions & 3 deletions ...cutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipelines/data_science/nodes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
import logging
from typing import Dict, Tuple

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.metrics import max_error, mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split


def split_data(data: pd.DataFrame, parameters: Dict) -> Tuple:
def split_data(data: pd.DataFrame, parameters: dict) -> tuple:
"""Splits data into features and targets training and test sets.

Args:
Expand Down Expand Up @@ -41,7 +40,7 @@ def train_model(X_train: pd.DataFrame, y_train: pd.Series) -> LinearRegression:

def evaluate_model(
regressor: LinearRegression, X_test: pd.DataFrame, y_test: pd.Series
) -> Dict[str, float]:
) -> dict[str, float]:
"""Calculates and logs the coefficient of determination.

Args:
Expand Down
2 changes: 1 addition & 1 deletion ...hts-pandas-viz/{{ cookiecutter.repo_name }}/tests/pipelines/data_science/test_pipeline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,4 +64,4 @@ def test_data_science_pipeline(caplog, dummy_data, dummy_parameters):

SequentialRunner().run(pipeline, catalog)

assert successful_run_msg in caplog.text
assert successful_run_msg in caplog.text
1 change: 0 additions & 1 deletion spaceflights-pandas/{{ cookiecutter.repo_name }}/docs/source/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-


# {{ cookiecutter.python_package }} documentation build
Expand Down
4 changes: 2 additions & 2 deletions spaceflights-pandas/{{ cookiecutter.repo_name }}/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ readme = "README.md"
dynamic = ["dependencies", "version"]

[project.scripts]
{{ cookiecutter.repo_name }} = "{{ cookiecutter.python_package }}.__main__:main"
"{{ cookiecutter.repo_name }}" = "{{ cookiecutter.python_package }}.__main__:main"

[project.entry-points."kedro.hooks"]

Expand Down Expand Up @@ -43,7 +43,7 @@ namespaces = false
package_name = "{{ cookiecutter.python_package }}"
project_name = "{{ cookiecutter.project_name }}"
kedro_init_version = "{{ cookiecutter.kedro_version }}"
tools = {{ cookiecutter.tools | default('') | string | replace('\"', '\\\"') }}
tools = "{{ cookiecutter.tools | default('') | string | replace('\"', '\\\"') }}"
example_pipeline = "{{ cookiecutter.example_pipeline }}"
source_dir = "src"

Expand Down
3 changes: 1 addition & 2 deletions ...s/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipeline_registry.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
"""Project pipelines."""
from typing import Dict

from kedro.framework.project import find_pipelines
from kedro.pipeline import Pipeline


def register_pipelines() -> Dict[str, Pipeline]:
def register_pipelines() -> dict[str, Pipeline]:
"""Register the project's pipelines.
Returns:
Expand Down
3 changes: 1 addition & 2 deletions ...cutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipelines/data_science/nodes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
import logging
from typing import Dict, Tuple

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split


def split_data(data: pd.DataFrame, parameters: Dict) -> Tuple:
def split_data(data: pd.DataFrame, parameters: dict) -> tuple:
"""Splits data into features and targets training and test sets.

Args:
Expand Down
2 changes: 1 addition & 1 deletion ...flights-pandas/{{ cookiecutter.repo_name }}/tests/pipelines/data_science/test_pipeline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,4 +64,4 @@ def test_data_science_pipeline(caplog, dummy_data, dummy_parameters):

SequentialRunner().run(pipeline, catalog)

assert successful_run_msg in caplog.text
assert successful_run_msg in caplog.text
1 change: 0 additions & 1 deletion spaceflights-pyspark-viz/{{ cookiecutter.repo_name }}/docs/source/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-


# {{ cookiecutter.python_package }} documentation build
Expand Down
4 changes: 2 additions & 2 deletions spaceflights-pyspark-viz/{{ cookiecutter.repo_name }}/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ readme = "README.md"
dynamic = ["dependencies", "version"]

[project.scripts]
{{ cookiecutter.repo_name }} = "{{ cookiecutter.python_package }}.__main__:main"
"{{ cookiecutter.repo_name }}" = "{{ cookiecutter.python_package }}.__main__:main"

[project.entry-points."kedro.hooks"]

Expand Down Expand Up @@ -43,7 +43,7 @@ namespaces = false
package_name = "{{ cookiecutter.python_package }}"
project_name = "{{ cookiecutter.project_name }}"
kedro_init_version = "{{ cookiecutter.kedro_version }}"
tools = {{ cookiecutter.tools | default('') | string | replace('\"', '\\\"') }}
tools = "{{ cookiecutter.tools | default('') | string | replace('\"', '\\\"') }}"
example_pipeline = "{{ cookiecutter.example_pipeline }}"
source_dir = "src"

Expand Down
3 changes: 1 addition & 2 deletions ...z/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipeline_registry.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
"""Project pipelines."""
from typing import Dict

from kedro.framework.project import find_pipelines
from kedro.pipeline import Pipeline


def register_pipelines() -> Dict[str, Pipeline]:
def register_pipelines() -> dict[str, Pipeline]:
"""Register the project's pipelines.
Returns:
Expand Down
3 changes: 1 addition & 2 deletions ...ter.repo_name }}/src/{{ cookiecutter.python_package }}/pipelines/data_processing/nodes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
from typing import Dict, Tuple

import pandas as pd
from pyspark.sql import Column
Expand All @@ -24,7 +23,7 @@ def _parse_money(x: Column) -> Column:
return x


def preprocess_companies(companies: SparkDataFrame) -> Tuple[SparkDataFrame, Dict]:
def preprocess_companies(companies: SparkDataFrame) -> tuple[SparkDataFrame, dict]:
"""Preprocesses the data for companies.

Args:
Expand Down
5 changes: 2 additions & 3 deletions ...cutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipelines/data_science/nodes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
import logging
from typing import Dict, Tuple

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.metrics import max_error, mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split


def split_data(data: pd.DataFrame, parameters: Dict) -> Tuple:
def split_data(data: pd.DataFrame, parameters: dict) -> tuple:
"""Splits data into features and targets training and test sets.

Args:
Expand Down Expand Up @@ -41,7 +40,7 @@ def train_model(X_train: pd.DataFrame, y_train: pd.Series) -> LinearRegression:

def evaluate_model(
regressor: LinearRegression, X_test: pd.DataFrame, y_test: pd.Series
) -> Dict[str, float]:
) -> dict[str, float]:
"""Calculates and logs the coefficient of determination.

Args:
Expand Down
2 changes: 1 addition & 1 deletion ...ts-pyspark-viz/{{ cookiecutter.repo_name }}/tests/pipelines/data_science/test_pipeline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,4 +64,4 @@ def test_data_science_pipeline(caplog, dummy_data, dummy_parameters):

SequentialRunner().run(pipeline, catalog)

assert successful_run_msg in caplog.text
assert successful_run_msg in caplog.text
1 change: 0 additions & 1 deletion spaceflights-pyspark/{{ cookiecutter.repo_name }}/docs/source/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-


# {{ cookiecutter.python_package }} documentation build
Expand Down
4 changes: 2 additions & 2 deletions spaceflights-pyspark/{{ cookiecutter.repo_name }}/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ readme = "README.md"
dynamic = ["dependencies", "version"]

[project.scripts]
{{ cookiecutter.repo_name }} = "{{ cookiecutter.python_package }}.__main__:main"
"{{ cookiecutter.repo_name }}" = "{{ cookiecutter.python_package }}.__main__:main"

[project.entry-points."kedro.hooks"]

Expand Down Expand Up @@ -43,7 +43,7 @@ namespaces = false
package_name = "{{ cookiecutter.python_package }}"
project_name = "{{ cookiecutter.project_name }}"
kedro_init_version = "{{ cookiecutter.kedro_version }}"
tools = {{ cookiecutter.tools | default('') | string | replace('\"', '\\\"') }}
tools = "{{ cookiecutter.tools | default('') | string | replace('\"', '\\\"') }}"
example_pipeline = "{{ cookiecutter.example_pipeline }}"
source_dir = "src"

Expand Down
3 changes: 1 addition & 2 deletions ...k/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipeline_registry.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
"""Project pipelines."""
from typing import Dict

from kedro.framework.project import find_pipelines
from kedro.pipeline import Pipeline


def register_pipelines() -> Dict[str, Pipeline]:
def register_pipelines() -> dict[str, Pipeline]:
"""Register the project's pipelines.
Returns:
Expand Down
3 changes: 1 addition & 2 deletions ...cutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipelines/data_science/nodes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
import logging
from typing import Dict, Tuple

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split


def split_data(data: pd.DataFrame, parameters: Dict) -> Tuple:
def split_data(data: pd.DataFrame, parameters: dict) -> tuple:
"""Splits data into features and targets training and test sets.

Args:
Expand Down
2 changes: 1 addition & 1 deletion ...lights-pyspark/{{ cookiecutter.repo_name }}/tests/pipelines/data_science/test_pipeline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,4 +64,4 @@ def test_data_science_pipeline(caplog, dummy_data, dummy_parameters):

SequentialRunner().run(pipeline, catalog)

assert successful_run_msg in caplog.text
assert successful_run_msg in caplog.text
Loading