v1.2.1

This is a bugfix release:

• #419 #409 Build fixes to enable the Conda build (and therefore publishing to conda-forge)
• #388 ArcticDB will now allow updates on data if the existing data was written prior to v1.1 regardless of whether that existing data is sorted or not

Full Changelog: v1.2.0...v1.2.1

ArcticDB distributes:

• wheels for Windows and Linux on PyPI, the latest being installable with:

  pip install --update arcticdb

• conda packages for MacOS and Linux on conda-forge, the latest being installable with:

  mamba install -c conda-forge --update arcticdb

Below are artifacts for debugging:

v1.2.0

🚀 Features

✅ defragment_symbol_data  method added (#180)

• This will defragment fragmented symbols. Fragmented symbols are typically caused by frequent small appends (e.g. 1 row every hour) and result in sub-optimal read performance. Reading defragmented symbols can result in significantly improved performance compared to a more fragmented equivalent!

✅ Write API for column-level statistics added. (#253)

• The structures created using this API will later be used to improve the performance of queries in later versions. As of version 1.2.0 however, the structures are not consumed at query time.

✅ Configuration variables set via environment variables are now case insensitive (#364)

🐛 Bug fixes

✅ add_to_snapshot  and remove_from_snapshot  no longer leave unreachable data on disk if a symbol-version pair being removed from a snapshot is the last reference to this version (#253)

✅ Reads with QueryBuilder parameters that contain a groupby/aggregation  clauses now only fetch data segments containing columns relevant to the query (#253)

Full Changelog: v1.1.0...v1.2.0

The wheels are on Pypi. Below are for debugging:

v1.1.0

New Platforms

✅ Version 1.1 includes the first official release of the Windows version of ArcticDB!
Versions 1.1 includes the following caveats:

• Please have a recent Visual C++ Redistributable for Visual Studio 2015-2022 installed.
• Windows does not support fixed-width NumPy strings and so will not be able to write fixed-width strings contained within a NumPy array.
• Writing pickled data can fail on machines with limited memory, see #340 for details and a workaround.
• The LMDB store on Windows pre-allocates the max storage size on disk, so we limited the size to 128MB. We will provide an option to increase the size in the future (#229).

✅ Version 1.1 now supports Python 3.11!

New Features

✅ batch_get_descriptor method added (#219)
✅ batch_get_metadata method added (#220)
Both batch methods do not achieve optimal parallelism - we are aiming to address in 1.2.

Changes

✅ 1.1 pins to Pandas < 2.0 whilst we address a few remaining compatibility issues 🐼 (#237)
✅ 1.1 supports Protobuf V3 as well as V4
✅ Out-of-order data updates now raises exceptions should they prevent future indexed reads (#203)
✅ PCRE is now statically linked (#321)
✅ stream id which is a confusing term has been renamed to symbol in error messages and should be better understood (#311)
✅ Pickled data supports up-to 4GB (#260)
✅ Debugging toolbox added (#209)
✅ Introduced a base exception type arcticdb.exceptions.ArcticException. Exceptions are exposed in arcticdb.exceptions module with the following hierarchy:

RuntimeError
└-- ArcticException
    |-- ArcticNativeNotYetImplemented
    |-- DuplicateKeyException
    |-- MissingDataException
    |-- NoDataFoundException
    |-- NoSuchVersionException
    |-- NormalizationException
    |-- PermissionException
    |-- SchemaException
    |-- SortingException
    |   └-- UnsortedDataException
    |-- StorageException
    |-- StreamDescriptorMismatch
    └-- InternalException

Next version:

NEW FEATURE: defragment_symbol_data method added (#180)

The wheels are on Pypi. Below are for debugging: