feat: support symbolic linked pipelines to avoid repetition

[daconstenla]

why

While using kpops and trying to reduce the amount of copy/pasting required to configure multi-tenant configuration I found that symbolic links would cause kpops execution to break.

what

Replaces existing Path.blob with blob.blob and resolves the real file location so no other changes are required for it to be compatible with existing logic.

Note

In python 3.13 symbolic links are supported docs.python.org/3.13#pathlib.Path.glob, probably it would be as easy as follows:
yield from sorted(pipeline_path.glob(f"**/{PIPELINE_YAML}", recurse_symlinks=True))

[disrupted]

Note

In python 3.13 symbolic links are supported docs.python.org/3.13#pathlib.Path.glob, probably it would be as easy as follows: yield from sorted(pipeline_path.glob(f"**/{PIPELINE_YAML}", recurse_symlinks=True))

that's a good head-up. how about a short todo comment in the code to refactor this once we drop support for older Python versions?

[disrupted]

please consider adding a test for it. If you need any help let me know

[daconstenla]

Note
In python 3.13 symbolic links are supported docs.python.org/3.13#pathlib.Path.glob, probably it would be as easy as follows: yield from sorted(pipeline_path.glob(f"**/{PIPELINE_YAML}", recurse_symlinks=True))

that's a good head-up. how about a short todo comment in the code to refactor this once we drop support for older Python versions?

Sure I can add a note, I actually had one originally but I moved it to the description of the pull-request.

[daconstenla]

please consider adding a test for it. If you need any help let me know

How would you suggest we test it? creating a pipeline that's just a symlink of another and expect same result? 🤔

[disrupted]

please consider adding a test for it. If you need any help let me know

How would you suggest we test it? creating a pipeline that's just a symlink of another and expect same result? 🤔

yes. feel free to commit the symlink to https://github.com/bakdata/kpops/tree/main/tests/pipeline/resources

[disrupted]

the new test itself would make sense in https://github.com/bakdata/kpops/blob/main/tests/pipeline/test_generate.py

edit: I believe ideally we would need to cover the following scenarios:

• kpops generate pipelines/regular-folder/symlinked-pipeline.yaml
• kpops generate pipelines/symlinked-folder
• kpops generate pipelines where pipelines contains symlinked folders

wdyt?

[daconstenla]

the new test itself would make sense in https://github.com/bakdata/kpops/blob/main/tests/pipeline/test_generate.py

edit: I believe ideally we would need to cover the following scenarios:

• kpops generate pipelines/regular-folder/symlinked-pipeline.yaml
• kpops generate pipelines/symlinked-folder
• kpops generate pipelines where pipelines contains symlinked folders

wdyt?

Mmmm, ok, more than I thought, let me try to cover those scenarios.

[daconstenla]

the new test itself would make sense in https://github.com/bakdata/kpops/blob/main/tests/pipeline/test_generate.py
edit: I believe ideally we would need to cover the following scenarios:

• kpops generate pipelines/regular-folder/symlinked-pipeline.yaml
• kpops generate pipelines/symlinked-folder
• kpops generate pipelines where pipelines contains symlinked folders

wdyt?

Mmmm, ok, more than I thought, let me try to cover those scenarios.

As suggested I've included 3 types of symlinks to the tests:

• symlinked pipeline
• symlinked folder
• folder with symlinked folder and symlinked pipeline as well as a regular pipeline

the test just validates the rendered results is as the original sources of the symlinks.

[disrupted]

LGTM, thanks for your contribution

[daconstenla]

@disrupted I assume you'll merge it when ready, thank you for the inputs.