-
Notifications
You must be signed in to change notification settings - Fork 6
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
6 changed files
with
29 additions
and
22 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
0.10.1 (2024-02-05) | ||
=================== | ||
|
||
Features | ||
-------- | ||
|
||
- Add support of ``Incremental Strategies`` for ``Kafka`` connection: | ||
|
||
.. code-block:: python | ||
reader = DBReader( | ||
connection=Kafka(...), | ||
source="topic_name", | ||
hwm=AutoDetectHWM(name="some_hwm_name", expression="offset"), | ||
) | ||
with IncrementalStrategy(): | ||
df = reader.run() | ||
This lets you resume reading data from a Kafka topic starting at the last committed offset from your previous run. (:github:pull:`202`) | ||
- Add ``has_data``, ``raise_if_no_data`` methods to ``DBReader`` class. (:github:pull:`203`) | ||
- Updare VMware Greenplum connector from ``2.1.4`` to ``2.3.0``. This implies: | ||
* Greenplum 7.x support | ||
* `Kubernetes support <https://docs.vmware.com/en/VMware-Greenplum-Connector-for-Apache-Spark/2.3/greenplum-connector-spark/configure.html#k8scfg>`_ | ||
* New read option `gpdb.matchDistributionPolicy <https://docs.vmware.com/en/VMware-Greenplum-Connector-for-Apache-Spark/2.3/greenplum-connector-spark/options.html#distpolmotion>`_ | ||
which allows to match each Spark executor with specific Greenplum segment, avoiding redundant data transfer between Greenplum segments | ||
* Allows overriding `Greenplum optimizer parameters <https://docs.vmware.com/en/VMware-Greenplum-Connector-for-Apache-Spark/2.3/greenplum-connector-spark/options.html#greenplum-gucs>`_ in read/write operations (:github:pull:`208`) | ||
- ``Greenplum.get_packages()`` method now accepts optional arg ``package_version`` which allows to override version of Greenplum connector package. (:github:pull:`208`) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -4,6 +4,7 @@ | |
|
||
DRAFT | ||
NEXT_RELEASE | ||
0.10.1 | ||
0.10.0 | ||
0.9.5 | ||
0.9.4 | ||
|
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.