Skip to content

Releases: tubular/sparkly

v2.3.0

25 Aug 09:32
Compare
Choose a tag to compare
  • Overwrite existing tables in the metastore.
  • Add functions module and provide switch_case column generation and multijoin.
  • Add implicit test target import and extended assertEqual variation.
  • Support writing to redis:// and rediss:// URLs.
  • Add LRU cache that persists DataFrames under the hood.
  • Add ability to check whether a complex type defines specific fields.

v2.2.1

03 Aug 10:22
Compare
Choose a tag to compare
  • spark.sql.shuffle.partitions in SparklyTest should be set to string,
    because int value breaks integration testing in Spark 2.0.2.

v2.2.0

03 Aug 09:31
Compare
Choose a tag to compare
  • Add instant iterative development mode. sparkly-testing --help for more details.
  • Use in-memory db for Hive Metastore in SparklyTest (faster tests).
  • spark.sql.shuffle.partitions = 4 for SparklyTest (faster tests).
  • spark.sql.warehouse.dir = <random tmp dir> for SparklyTest (no side effects)

v2.1.1

27 Jul 11:49
Compare
Choose a tag to compare
  • Fix: remove backtick quoting from catalog utils to ease work with different databases.

v2.1.0

10 Jul 11:49
Compare
Choose a tag to compare
  • Add ability to specify custom maven repositories.

v2.0.4

10 Jul 11:49
Compare
Choose a tag to compare
  • Make it possible to override default value of spark.sql.catalogImplementation

v2.0.3

10 Jul 11:48
Compare
Choose a tag to compare
  • Add KafkaWatcher to facilitate testing of writing to Kafka
  • Fix a few minor pyflakes warnings and typos

v2.0.2

10 Jul 11:48
Compare
Choose a tag to compare
  • Fix: #40 write_ext.kafka ignores errors.

v2.0.1

13 Feb 15:56
Compare
Choose a tag to compare
  • Migrate to Spark 2, Spark 1.6.x isn't supported by sparkly 2.x.
  • Rename SparklyContext to SparklySession and derive it from SparkSession.
  • Use built-in csv reader.
  • Replace hms with catalog_ext.
  • parse_schema is now consistent with DataType.simpleString method.

v1.1.1

13 Feb 10:56
Compare
Choose a tag to compare
  • Fix: kafka import error.