Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python Bindings: Numpy Support #29

Merged
merged 42 commits into from
Apr 18, 2019
Merged
Changes from 1 commit
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
323c410
python: include scikit-build cmake scripts
SteVwonder Dec 10, 2018
ca5097b
python: add initial python bindings with minimal numpy support
SteVwonder Dec 10, 2018
655b5f4
python: add test for bindings
SteVwonder Dec 10, 2018
f9c4a5c
python: add FindNumPy cmake module
SteVwonder Dec 23, 2018
ccf5f4f
python: add error checking around decompress and malloc
SteVwonder Dec 28, 2018
fb1fa61
travis: add BUILD_PYTHON to build matrix
SteVwonder Dec 23, 2018
61c4f23
python: add support for fixed-rate and fixed-precision modes
SteVwonder Dec 28, 2018
16237e9
python: move tests from python to tests/python
SteVwonder Dec 28, 2018
86193ce
python: avoid use of reserved keywords
SteVwonder Dec 31, 2018
2d5c133
travis: switch python tests to ubuntu xenial from trusty
SteVwonder Dec 31, 2018
573cb09
python: split zfp into pxd header and pyx implementation files
SteVwonder Mar 3, 2019
f9e2874
python: remove PY2 reference in cmake
SteVwonder Mar 3, 2019
b1f4151
python: add `write_header` option to `compress_numpy`
SteVwonder Mar 13, 2019
41f215f
python: re-enable integer types in `compress_numpy`
SteVwonder Mar 13, 2019
cc26739
python: export zfp_mode and zfp_type enum values
SteVwonder Mar 13, 2019
ab64253
python/test_utils: add initial bindings for test utilities
SteVwonder Mar 13, 2019
f15669e
python/test_numpy: add checksumming tests via the test_utils
SteVwonder Mar 13, 2019
32d599f
python/test_utils: add input argument validation
SteVwonder Mar 13, 2019
82c86b1
python: replace fortran-contig transformation with setting strides
SteVwonder Mar 14, 2019
9106f2f
tests/python: only use fixed-rate compression on integer types
SteVwonder Mar 14, 2019
5641ce8
python: add decompression into user-supplied buffer
SteVwonder Mar 24, 2019
e1704a6
tests/python: replace ad-hoc round-trip tests with checksum tests
SteVwonder Mar 24, 2019
667092b
python: minor formatting tweaks and cleanup
SteVwonder Mar 25, 2019
873c0a2
python: refactor decompress to simplify the top-level function
SteVwonder Mar 25, 2019
e142066
python/cmake: include function signatures and docstrings
SteVwonder Mar 25, 2019
b3f7a1b
python: use nogil to release GIL when (de-)compressing
SteVwonder Mar 25, 2019
08d717e
python: check the return of zfp_write_header
SteVwonder Mar 25, 2019
c225e3a
python: remove reference to long long for C89 compliance
SteVwonder Mar 25, 2019
ffffa37
finalize travis CI config for python testing: default 3.5 over differ…
salasoom Mar 25, 2019
df52af4
python: split decomp into simple "public" and complex "private" APIs
SteVwonder Mar 28, 2019
4870524
python: add support for lossless compression
SteVwonder Mar 28, 2019
e361949
python: reverse stride order to optimize for default C-layout
SteVwonder Mar 28, 2019
9f2436f
docs: add initial python bindings documentation
SteVwonder Mar 30, 2019
a1285d2
python: fix checksum tests for permuted and interleaved
SteVwonder Apr 1, 2019
fb127f6
python: remove dependency on the 'six' python package
SteVwonder Apr 1, 2019
657b140
python: add requirements.txt for py dependencies
SteVwonder Apr 1, 2019
00c8133
travis: attempt to fix python builds
SteVwonder Apr 1, 2019
6d68fb3
python: reverse shapes and strides in `_decompress`
SteVwonder Apr 4, 2019
90b1d7d
python: throw exception if `zfp_compress` returns 0
SteVwonder Apr 4, 2019
7c11b63
docs/python: add py:module:: zfp
SteVwonder Apr 4, 2019
8d612d8
refactor _decompress to assume that no header is present
SteVwonder Apr 8, 2019
8d64d04
python: re-add comparison of output metadata against user-provided
SteVwonder Apr 8, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
python: add support for lossless compression
SteVwonder authored and salasoom committed Apr 17, 2019
commit 4870524615d2b2c20fc83f14947168227a4593ba
1 change: 1 addition & 0 deletions python/zfp.pxd
Original file line number Diff line number Diff line change
@@ -46,6 +46,7 @@ cdef extern from "zfp.h":
cython.uint zfp_stream_set_precision(zfp_stream* stream, cython.uint precision);
double zfp_stream_set_accuracy(zfp_stream* stream, double tolerance);
double zfp_stream_set_rate(zfp_stream* stream, double rate, zfp_type type, cython.uint dims, int wra);
void zfp_stream_set_reversible(zfp_stream* stream);
stdint.uint64_t zfp_stream_mode(const zfp_stream* zfp);
zfp_mode zfp_stream_set_mode(zfp_stream* stream, stdint.uint64_t mode);
zfp_field* zfp_field_alloc();
6 changes: 2 additions & 4 deletions python/zfp.pyx
Original file line number Diff line number Diff line change
@@ -135,9 +135,7 @@ cpdef bytes compress_numpy(
if arr is None:
raise TypeError("Input array cannot be None")
num_params_set = sum([1 for x in [tolerance, rate, precision] if x >= 0])
if num_params_set == 0:
raise ValueError("Either tolerance, rate, or precision must be set")
elif num_params_set > 1:
if num_params_set > 1:
raise ValueError("Only one of tolerance, rate, or precision can be set")

# Setup zfp structs to begin compression
@@ -223,7 +221,7 @@ cdef _set_compression_mode(
elif precision >= 0:
zfp_stream_set_precision(stream, precision)
else:
raise ValueError("Either tolerance, rate, or precision must be set")
zfp_stream_set_reversible(stream)

cdef _validate_4d_list(in_list, list_name):
# Validate that the input list is either a valid list for strides or shape
37 changes: 37 additions & 0 deletions tests/python/test_numpy.py
Original file line number Diff line number Diff line change
@@ -5,9 +5,46 @@
import zfp
import test_utils
import numpy as np
try:
from packaging.version import parse as version_parse
except ImportError:
version_parse = None


class TestNumpy(unittest.TestCase):
def lossless_round_trip(self, orig_array):
compressed_array = zfp.compress_numpy(orig_array, write_header=True)
decompressed_array = zfp.decompress_numpy(compressed_array)
self.assertIsNone(np.testing.assert_array_equal(decompressed_array, orig_array))

def test_different_dimensions(self):
for dimensions in range(1, 5):
shape = [5] * dimensions

c_array = np.random.rand(*shape)
self.lossless_round_trip(c_array)


def test_different_dtypes(self):
shape = (5, 5)
num_elements = shape[0] * shape[1]

for dtype in [np.float32, np.float64]:
elements = np.random.random_sample(num_elements)
elements = elements.astype(dtype, casting="same_kind")
array = np.reshape(elements, newshape=shape)
self.lossless_round_trip(array)

if (version_parse is not None and
(version_parse(np.__version__) >= version_parse("1.11.0"))
):
for dtype in [np.int32, np.int64]:
array = np.random.randint(2**30, size=shape, dtype=dtype)
self.lossless_round_trip(array)
else:
array = np.random.randint(2**30, size=shape)
self.lossless_round_trip(array)

def test_utils(self):
for ndims in range(1, 5):
for ztype, ztype_str in [