CHANGELOG.md

Change Log

Unreleased - ReleaseDate

0.5.8 - 2024-08-14

• Separate qualifier for new and old sync pks (splitgraph#608)
• Allow running Seafowl without an object store / catalog config. (splitgraph#607)
• Remove bucket from options (splitgraph#603)
• Remove prefix from options (splitgraph#595)
• Migrate unstable field json repr to a canonical one (splitgraph#594)
• Add flag denoting first sync response (splitgraph#593)
• Refactor object_store_factory and create StorageLocationInfo (splitgraph#592)
• Support empty syncs with no batches provided as payload (splitgraph#591)
• Add origin and position to flight handler debug (splitgraph#590)
• Patch DF to circumvent an analyzer rule (splitgraph#586)
• DataFusion 40 upgrade (splitgraph#584)
• Optional sync sequence number (splitgraph#581)
• BEAC-167: Move object store configs to object_store_factory (splitgraph#580)
• Add criteria for max separate sync calls to apply when flushing (splitgraph#579)
• Add memory and local FS store to the object_store_factory builder function (splitgraph#578)
• BEAC-167: Create a crate that instantiates an object store based on URL, settings and environment variables (splitgraph#569)
• TPC-DS Q75 fix backport (splitgraph#563)
• Workflow_dispatch trigger for nightly workflow (splitgraph#562)
• Allow using RUST_LOG_STYLE=never to disable ANSI colors (splitgraph#561)
• Use the storage location name as the key in the clade schema (splitgraph#556)
• Increase retries and add delay backoff to connect to client on tests (splitgraph#554)
• Make origin a string (splitgraph#548)

0.5.7 - 2024-06-21

• Add flush task (splitgraph#534)
• Add sync metrics (splitgraph#533)
• Upgrade to DataFusion 39 (splitgraph#532)
• Add support for inline-ing metastore over Flight (splitgraph#531)
• Sync protocol improvements (splitgraph#530)
• Data replication interface (splitgraph#527)
• Autodetect bucket region in S3 without custom endpoints (splitgraph#526)
• Ignore DeltaTableAlready delta error during conversion of a parquet folder (splitgraph#525)
• Make CONVERT TO DELTA overwrite if exists (splitgraph#523)
• DataFusion 37.1 upgrade (splitgraph#520)
• Add various object store cache Prometheus metrics (splitgraph#519)
• Add Prometheus metrics to MemoryManager (splitgraph#518)

0.5.6 - 2024-04-22

• Improve Arrow Flight SQL interface logging (splitgraph#517)
• Move Delta store creation into a separate ObjectStoreFactory (splitgraph#516)
• Best effort chunk coalescing implementation (splitgraph#515)
• Use tokio-graceful-shutdown for SIGINT / SIGTERM (splitgraph#513)
• Upgrade to DataFusion 36 (splitgraph#511)
• Enable using session tokens for the S3 client (splitgraph#510)
• Fix condition for caching clade object stores (splitgraph#509)
• Add caching on top of clade-provided object stores too (splitgraph#508)
• Improve caching object store performance (splitgraph#507)
• Make object_store confgi section optional (splitgraph#506)
• Add support for dynamic object store usage via clade (splitgraph#504)

0.5.5 - 2024-02-26

• Add a http health endpoint (splitgraph#502)
• Add support for public S3 buckets (splitgraph#501)
• Add gRPC metrics (splitgraph#500)
• Implement prometheus metrics (splitgraph#499)
• Upgrade to DataFusion 35 and Arrow 50 (splitgraph#498)
• Replace log with tracing (splitgraph#493)
• Use from_env when creating the DataFusion session config (splitgraph#490)

0.5.4 - 2024-01-17

• Upgrade to DataFusion 34 and Arrow 49 (splitgraph#486)
• Enable external metastores (splitgraph#483)
• Eager dropping of tables and schemas (splitgraph#481)
• Add Arrow Flight SQL frontend (splitgraph#478)

0.5.3 - 2023-12-04

• Implement CONVERT TO DELTA statement (splitgraph#473)
• Support sub-folder paths as data firs in object stores (splitgraph#472)
• Introduce Seafowl CLI (splitgraph#468)
• Upgrade to DataFusion 32 (splitgraph#465)

0.5.2 - 2023-09-07

• Wire up COPY TO statement (splitgraph#462)
• Upgrade to DataFusion 30 and Arrow 45 (splitgraph#461)
• Enable time travel for write statements with source queries (splitgraph#460)

0.5.1 - 2023-08-01

• Upgrade to DataFusion 28 and Arrow 43 (splitgraph#458)
• Implement CREATE OR REPLACE FUNCTION statement path (splitgraph#455)
• Enable passing of supplied get options via the HTTP object store (splitgraph#454)

0.5.0 - 2023-07-12

• Upgrade to DataFusion 27 and Arrow 42 (splitgraph#453)
• Load schemas without tables and double-check missing DB ids in a given context (splitgraph#446)
• Implement schema coercion and file streaming in the upload endpoint (splitgraph#439)
• Return timing data in response headers (splitgraph#438)

0.4.3 - 2023-06-07

• Enable creating external tables on cloud object stores (splitgraph#433)
• Implement streaming of result rows (splitgraph#424)

0.4.2 - 2023-05-26

• Enable specifying the GET endpoint cache-control header via config (splitgraph#422)
• Upgrade to DataFusion 25 and Arrow 39 (splitgraph#415)

0.4.1 - 2023-05-17

• Upgrade to DataFusion 24 and Arrow 38 (splitgraph#403)
• Enable passing query as a path parameter in the GET endpoint (splitgraph#402)

0.4.0 - 2023-05-11

• Remove legacy table reading logic (splitgraph#390)
• Upgrade DataFusion to 23 and Arrow to 37 (splitgraph#386)
• Add support for GCS bucket object stores (splitgraph#379)

0.3.4 - 2023-04-27

• Return result type info in the content-type header, and bump object_store crate (splitgraph#367)

0.3.3 - 2023-04-20

• Fix table renaming to quoted identifiers with special characters (splitgraph#362)
• Upgrade DataFusion to v22.0.0 and Arrow to v36.0.0 (splitgraph#360)

0.3.2 - 2023-04-04

• Enable caching of S3 objects in the local FS (splitgraph#341)
• Fix the silent panic in the cache eviction hook rendering it useless (splitgraph#334)

0.3.1 - 2023-03-29

• Fix unimplemented method in AWS object store, and broaden integration test coverage by extending target object stores to include a MinIO instance (splitgraph#331)
• Enable cached GET endpoint for authorized requests, and enable dropping of external tables (splitgraph#326)

0.3.0 - 2023-03-22

• Migration of storage layer to delta-rs (splitgraph#307)
• Purge legacy logic (splitgraph#313)
• Deltify parquet writing logic (splitgraph#316)
• Implement VACUUM DATABASE command (splitgraph#320)
• Enable VACUUM for Delta tables (splitgraph#322)

0.2.12 - 2023-02-14

• Upgrade to DataFusion 18.0.0 (splitgraph#297)
• Implement multi-database support for upload endpoint (splitgraph#291)
• Implement multi-database support for GET and POST query endpoints (splitgraph#289)

0.2.11 - 2023-01-27

• Upgrade to DataFusion post-16.0 (splitgraph#270)
• Add an ssl_cert_file option to the config (splitgraph#278)
• Migrate to DataFusion's native DML nodes for logical write statement planning (splitgraph#281)

0.2.10 - 2022-12-30

• Add region config parameter for Amazon S3 object store (splitgraph#255)
• Enable querying external Delta tables in Seafowl (splitgraph#252)
• Implement remote table factory (splitgraph#250)

0.2.9 - 2022-12-23

• Add support for pushdown in remote tables:
  • LIMIT (splitgraph#221)
  • WHERE (splitgraph#226, splitgraph#235)
• Factor out remote tables into a separate crate (splitgraph#238)
• Upgrade to DataFusion 15 (splitgraph#248)

0.2.8 - 2022-11-21

• Implement table_partitions system table (splitgraph#214)
• Add WASI + MessagePack UDF language variant (splitgraph#149)

0.2.7 - 2022-11-17

• Import JSON values as strings in CREATE EXTERNAL TABLE (splitgraph#208)
• Add support for SQLite in CREATE EXTERNAL TABLE (splitgraph#200)

0.2.6 - 2022-11-08

• Update to DataFusion 14 / Arrow 26 (splitgraph#198)
• Bugfix: VACUUM with shared partitions (splitgraph#191)
• Bugfix: DELETE with certain filters that cover a whole partition (splitgraph#192)
• Initial support for other databases in CREATE EXTERNAL TABLE (splitgraph#190)
  • More documentation pending. Example: CREATE EXTERNAL TABLE t STORED AS TABLE 'public.t' LOCATION 'postgresql://uname:pw@host:port/dbname'

0.2.5 - 2022-11-02

• Upgrade to DataFusion 13 (784f10bb) / Arrow 25.0.0 (splitgraph#176)
• Use ZSTD compression in Parquet files (splitgraph#182)
• Fix HTTP external tables using pre-signed S3 URLs (splitgraph#183)
• Fix INSERT INTO .. SELECT FROM (splitgraph#184)
• Fix some OUTER JOIN issues by using a minimum of 2 target_partitions (splitgraph#189)

0.2.4 - 2022-10-25

• Add system.table_versions table for inspecting table history (splitgraph#168)
• Add SQLite catalog.read_only option for compatibility with LiteFS replicas (splitgraph#171)

0.2.3 - 2022-10-21

• Add support for time travel queries (SELECT * FROM table('2022-01-01T00:00:00')) (splitgraph#154)
• Allow overriding SQLite journal mode (splitgraph#158)

0.2.2 - 2022-10-12

• Allow using SQL types in WASM UDF definitions (splitgraph#147)

0.2.1 - 2022-09-30

• Cached GET API now accepts URL-encoded query text in X-Seafowl-Header (splitgraph#122)
• Add support for DELETE statements (splitgraph#121)
• Add support for UPDATE statements (splitgraph#127)

0.2.0 - 2022-09-21

Breaking: Previous versions of Seafowl won't be able to read data written by this version.

• Fix storage of nullable / list columns (splitgraph#119)
• Default columns in CREATE TABLE to nullable (splitgraph#119)

0.1.1 - 2022-09-16

• Upgrade to DataFusion 12 (splitgraph#113)

0.1.0 - 2022-09-12

Fixes

• Use multi-part uploads to fix the memory usage issue when uploading data to S3 (splitgraph#99)