Versatile Data Kit 1.4

[antoniivanov]

Major features include:

Control Service

Complete Data Job Configuration Persistence (Pre-alpha)

The current two-step process of storing data job deployment configurations in both Kubernetes and a database leads to performance degradation, potential data loss, and complexity; optimizing storage by consistently keeping all essential properties in the database can enhance efficiency, system reliability, and user experience

Another important benefit would be to allow to track deployment status using the API.

vdk-structlog log plugin

The plugin allows users to configure logging metadata and logging format. It also works with bound loggers.

This plugin allows users to:

select the log output format
configure the logging metadata
display metadata added by bound loggers

See more in its documentation page

vdk-core Error handling changes

Deprecated error reporting patterns

Most vdk-core generic Exceptions replaced with Domain specific

What's Changed

• control-service: Enhance Exception Handling for DataJobsSynchronizer by @mivanov1988 in control-service: Enhance Exception Handling for DataJobsSynchronizer #2758
• control-service: [bug fix] Add freetype2 and libpng to secure builder by @doks5 in control-service: [bug fix] Add freetype2 and libpng to secure builder #2744
• control-service: add IT tests for async job deployment by @mivanov1988 in control-service: add IT tests for async job deployment #2794
• control-service: add new deployment tables by @mivanov1988 in control-service: add new deployment tables #2719
• control-service: asynchronous deployment deletion by @mivanov1988 in control-service: asynchronous deployment deletion #2781
• control-service: data job synchronizer error handling by @mivanov1988 in control-service: data job synchronizer error handling #2742
• control-service: deployment controller reads from db by @mrMoZ1 in control-service: deployment controller reads from db #2800
• control-service: depoyment controller writes deployment entity by @mrMoZ1 in control-service: depoyment controller writes deployment entity #2731
• control-service: enable scheduled execution for data jobs' synchronizer by @mivanov1988 in control-service: enable scheduled execution for data jobs' synchronizer #2771
• control-service: fix control service post deployment test by @mivanov1988 in control-service: fix control service post deployment test #2790
• control-service: fix data job image building by @mivanov1988 in control-service: fix data job image building #2832
• control-service: fix infinite redeployment by @mivanov1988 in control-service: fix infinite redeployment #2822
• control-service: fix post deployment test by @mrMoZ1 in control-service: fix post deployment test #2815
• control-service: fix read deployment job version by @mivanov1988 in control-service: fix read deployment job version #2819
• control-service: handle deployment deletion in case of a job being deleted by @mivanov1988 in control-service: handle deployment deletion in case of a job being deleted #2816
• control-service: implement multi-threading for synchronization process by @mivanov1988 in control-service: implement multi-threading for synchronization process #2775
• control-service: improve async deployment logging by @mivanov1988 in control-service: improve async deployment logging #2826
• control-service: job resources validation on job deployment by @mivanov1988 in control-service: job resources validation on job deployment #2793
• control-service: reduce logging by @mivanov1988 in control-service: reduce logging #2834
• control-service: resolve dependabot alert by @antoniivanov in control-service: resolve dependabot alert #2751
• control-service: user-initiated deployment notifications by @mivanov1988 in control-service: user-initiated deployment notifications #2757
• control-service: utilize new deployment tables by @mivanov1988 in control-service: utilize new deployment tables #2714
• vdk-audit: Clean up some audit events by @doks5 in vdk-audit: Clean up some audit events #2792
• vdk-control-cli: fix CI/CD tests by @yonitoo in vdk-control-cli: fix CI/CD tests #2782
• vdk-control-cli: pin werkzeug to version 2.3.8 or less by @DeltaMichael in vdk-control-cli: pin werkzeug to version 2.3.8 or less #2743
• vdk-core: add error formatter configuration by @DeltaMichael in vdk-core: add error formatter configuration #2754
• vdk-core: create ingestion exceptions by @antoniivanov in vdk-core: create ingestion exceptions #2752
• vdk-core: domain specific properties/secrets exceptions by @antoniivanov in vdk-core: domain specific properties/secrets exceptions #2770
• vdk-core: fix postgres and greenplum tests by @yonitoo in vdk-core: fix postgres and greenplum tests #2825
• vdk-core: move error classifying logic by @duyguHsnHsn in vdk-core: move error classifying logic  #2769
• vdk-core: pass exceptions from data job steps in results by @DeltaMichael in vdk-core: pass exceptions from data job steps in results #2774
• vdk-core: remove code duplication in ingestion router by @antoniivanov in vdk-core: remove code duplication in ingestion router #2760
• vdk-core: simplify error message for send_**_for_ingestion by @antoniivanov in vdk-core: simplify error message for send_**_for_ingestion #2787
• vdk-core: test exception propagation to user code by @DeltaMichael in vdk-core: test exception propagation to user code #2820
• vdk-core: test ingestion with multiple threads by @antoniivanov in vdk-core: test ingestion with multiple threads #2796
• vdk-core: tests passing custom iterator to ingestion methods by @antoniivanov in vdk-core: tests passing custom iterator to ingestion methods #2761
• vdk-coverity: Adding Coverity Scan by @shanmathik in vdk-coverity: Adding Coverity Scan  #2753
• vdk-dag, vdk-control-cli, airflow-provider-vdk: step using deprecated field by @antoniivanov in vdk-dag, vdk-control-cli, airflow-provider-vdk: step using deprecated field #2706
• vdk-dag: fix failing validation tests by @DeltaMichael in vdk-dag: fix failing validation tests #2712
• vdk-heartbeat: Introduce additional sleep when checking deployments by @doks5 in vdk-heartbeat: Introduce additional sleep when checking deployments #2824
• vdk-impala: add Out Of Memory error handling by @dakodakov in vdk-impala: add Out Of Memory error handling #2747
• vdk-impala: introduce new error handling by @duyguHsnHsn in vdk-impala: introduce new error handling #2759
• vdk-jupyter: Enable PYTHONUNBUFFERED to ensure correct log ordering by @gageorgiev in vdk-jupyter: Enable PYTHONUNBUFFERED to ensure correct log ordering #2711
• vdk-jupyter: add Tutorial link in getting-started.ipynb by @antoniivanov in vdk-jupyter: add Tutorial link in getting-started.ipynb #2707
• vdk-jupyter: cicd fix by @duyguHsnHsn in vdk-jupyter: cicd fix #2780
• vdk-jupyter: fix bug in detecting run functions by @antoniivanov in vdk-jupyter: fix bug in detecting run functions #2721
• vdk-jupyter: fix ci/cd by @duyguHsnHsn in vdk-jupyter: fix ci/cd  #2773
• vdk-jupyter: print summary output to temp dir by @antoniivanov in vdk-jupyter: print summary output to temp dir #2715
• vdk-jupyter: update getting started to incldue vdksql by @antoniivanov in vdk-jupyter: update getting started to incldue vdksql #2713
• vdk-jupyter: use vdksql for SQL cells and steps by @antoniivanov in vdk-jupyter: use vdksql for SQL cells and steps #2729
• vdk-notebook: [bug fix] ignore missing id field in cell by @doks5 in vdk-notebook: [bug fix] ignore missing id field in cell #2717
• vdk-notebook: remove obsolete code by @antoniivanov in vdk-notebook: remove obsolete code #2716
• vdk-notebook: set summary file path as configuration by @antoniivanov in vdk-notebook: set summary file path as configuration #2709
• vdk-plugins: add new error handling methods by @duyguHsnHsn in vdk-plugins: add new error handling methods #2750
• vdk-structlog: create structured logging plugin by @DeltaMichael in vdk-structlog: create structured logging plugin #2801
• vdk-test-utils: make IngestIntoMemoryPlugin method configurable by @antoniivanov in vdk-test-utils: make IngestIntoMemoryPlugin method configurable #2783

New Contributors

• @shanmathik made their first contribution in vdk-coverity: Adding Coverity Scan  #2753

Full Changelog: v1.3...v1.4

---

This discussion was created from the release Versatile Data Kit 1.4.