Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Zstd compressor (#3) #1604

Open
wants to merge 38 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
0689612
Zstd compressor (#3)
clusty Dec 21, 2023
574988e
Fix single blob
clusty Jan 8, 2024
0e90ff1
Add license headers. Clang-format example file.
pleprince Jan 10, 2024
057dd80
Build Blosc as a part of the regular build (#4)
clusty Jan 15, 2024
78ca6c6
Implement the compressor inside OpenEXRCore (#5)
clusty Jan 22, 2024
41afe2e
Fix single blob
clusty Jan 8, 2024
f52266d
Add license headers. Clang-format example file.
pleprince Jan 10, 2024
d0a24d4
Build Blosc as a part of the regular build (#4)
clusty Jan 15, 2024
1e6698e
Implement the compressor inside OpenEXRCore (#5)
clusty Jan 22, 2024
1d2cd0c
Merge branch 'main' of https://github.com/clusty/openexr into main
pleprince Feb 20, 2024
48fa372
Merge remote-tracking branch 'aswf/main' into main
pleprince Mar 13, 2024
ccac098
Adopt new compression API
pleprince Mar 13, 2024
2341fb4
Fix API tests
pleprince Mar 13, 2024
6cfc3cc
Fix cmake build
pleprince Mar 13, 2024
21c60db
Merge latest from aswf main
pleprince Apr 2, 2024
88c0348
fix: debug build was not linking against debug blosc lib.
pleprince Apr 2, 2024
05d9457
fix: debug build was not linking against debug blosc lib, for older c…
pleprince Apr 2, 2024
9799bf9
fix: yet another variation on the same theme
pleprince Apr 2, 2024
a06daec
fix: debug and static builds should work.
pleprince Apr 3, 2024
2d678ae
fix: update bazel build
pleprince Apr 3, 2024
2ad98fe
fix: potential bazel typo
pleprince Apr 3, 2024
f2bdfde
fix: refactored linking and fixed validation
pleprince Apr 4, 2024
a887716
fix: update bazel blosc version
pleprince Apr 4, 2024
688d0d6
fix: tentative: always enable threads to build examples as it is need…
pleprince Apr 5, 2024
387d2ca
fix: Replacing "dl" with CMAKE_DL_LIBS to fix windows build
pleprince Apr 5, 2024
06f30a8
fix: update linking for CI fuzzing
pleprince Apr 5, 2024
903c2c4
fix(bazel): add missing source files to BUILD.bazel
pleprince Apr 5, 2024
40b1937
fix(bazel): tentative: add blosc3 deps to example build
pleprince Apr 5, 2024
bb8754a
fix(bazel): tentative: add linkopts to example build
pleprince Apr 5, 2024
15844b5
fix(bazel): tentative: add linkopts+deps to example build
pleprince Apr 5, 2024
a9ebf07
fix(bazel): everything but the examples build. Can't make it work loc…
pleprince Apr 5, 2024
55f37b0
fix(fuzz): tentative: use target name to link
pleprince Apr 5, 2024
4c714a2
build(bazel): switch c-blosc2 tp 2.12.0.bcr.2
pleprince Apr 23, 2024
81e1701
build(fuzz): build fuzz with dwarf-4. (test)
pleprince Apr 23, 2024
5b0be1d
build(fuzz): build fuzz with dwarf-4. (test #2)
pleprince Apr 23, 2024
cb6a1e0
build(fuzz): build fuzz with dwarf-4. (test #3)
pleprince Apr 23, 2024
17a3fdf
Merge branch 'main' of https://github.com/AcademySoftwareFoundation/o…
pleprince Jun 5, 2024
5047f7f
ci: temporarily disable fuzz test
pleprince Jun 19, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/ossfuzz_workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,12 @@ on:
- '!src/wrappers/**'
- '!.github/workflows/**'
- '.github/workflows/ossfuzz_workflow.yml'
# FIXME: Fuzzing build fails because we need to add libblosc2 to the list
# of linked libs in:
# https://github.com/google/oss-fuzz/blob/master/projects/openexr/build.sh
# ... but we can only do it once this PR is merged.
branches-ignore: # FIXME: REMOVE THIS CONDITION BEFORE MERGING !
- 'main'

permissions:
contents: read
Expand Down
7 changes: 7 additions & 0 deletions BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,7 @@ cc_library(
"src/lib/OpenEXRCore/internal_win32_file_impl.h",
"src/lib/OpenEXRCore/internal_xdr.h",
"src/lib/OpenEXRCore/internal_zip.c",
"src/lib/OpenEXRCore/internal_zstd.c",
"src/lib/OpenEXRCore/memory.c",
"src/lib/OpenEXRCore/opaque.c",
"src/lib/OpenEXRCore/openexr_version.h",
Expand Down Expand Up @@ -253,10 +254,12 @@ cc_library(
":windows": [],
"//conditions:default": [
"-pthread",
"-ldl",
],
}),
visibility = ["//visibility:public"],
deps = [
"@c-blosc2//:c-blosc2",
"@imath//:Imath",
"@libdeflate//:deflate",
],
Expand Down Expand Up @@ -357,6 +360,7 @@ cc_library(
"src/lib/OpenEXR/ImfWav.cpp",
"src/lib/OpenEXR/ImfZip.cpp",
"src/lib/OpenEXR/ImfZipCompressor.cpp",
"src/lib/OpenEXR/ImfZstdCompressor.cpp",
"src/lib/OpenEXR/b44ExpLogTable.h",
"src/lib/OpenEXR/dwaLookups.h",
],
Expand Down Expand Up @@ -476,6 +480,7 @@ cc_library(
"src/lib/OpenEXR/ImfXdr.h",
"src/lib/OpenEXR/ImfZip.h",
"src/lib/OpenEXR/ImfZipCompressor.h",
"src/lib/OpenEXR/ImfZstdCompressor.h",
"src/lib/OpenEXR/OpenEXRConfig.h",
"src/lib/OpenEXR/OpenEXRConfigInternal.h",
],
Expand All @@ -495,10 +500,12 @@ cc_library(
":windows": [],
"//conditions:default": [
"-pthread",
"-ldl",
],
}),
visibility = ["//visibility:public"],
deps = [
"@c-blosc2//:c-blosc2",
":IlmThread",
":OpenEXRCore",
"@imath//:Imath",
Expand Down
1 change: 1 addition & 0 deletions MODULE.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@ bazel_dep(name = "bazel_skylib", version = "1.6.1")
bazel_dep(name = "imath", version = "3.1.11")
bazel_dep(name = "libdeflate", version = "1.20.bcr.1")
bazel_dep(name = "platforms", version = "0.0.10")
bazel_dep(name = "c-blosc2", version = "2.12.0.bcr.2")
1 change: 1 addition & 0 deletions cmake/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,7 @@ if(OPENEXR_INSTALL_PKG_CONFIG)
function(openexr_pkg_config_help pcinfile)
string(TOUPPER "${CMAKE_BUILD_TYPE}" uppercase_CMAKE_BUILD_TYPE)
set(LIB_SUFFIX_DASH ${OPENEXR_LIB_SUFFIX}${CMAKE_${uppercase_CMAKE_BUILD_TYPE}_POSTFIX})
set(LIB_BUILD_SUFFIX ${CMAKE_${uppercase_CMAKE_BUILD_TYPE}_POSTFIX})
if(OPENEXR_ENABLE_THREADING AND TARGET Threads::Threads)
# hrm, can't use properties as they end up as generator expressions
# which don't seem to evaluate
Expand Down
14 changes: 14 additions & 0 deletions cmake/LibraryDefine.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,20 @@ function(OPENEXR_DEFINE_LIBRARY libname)
# we are embedding libdeflate
target_include_directories(${objlib} PRIVATE ${EXR_DEFLATE_INCLUDE_DIR})

# we are statically linking blosc2
if(${objlib} STREQUAL "OpenEXR" OR ${objlib} STREQUAL "OpenEXRCore")
message(STATUS "Blosc2: setting up for ${objlib}...")
message(STATUS ">> BLOSC2_INCLUDE_DIRS: ${BLOSC2_INCLUDE_DIRS}")
message(STATUS ">> BLOSC2_LIB_DIR: ${BLOSC2_LIB_DIR}")
target_include_directories(${objlib} PRIVATE ${BLOSC2_INCLUDE_DIRS})
target_link_directories(${objlib} PRIVATE ${BLOSC2_LIB_DIR})
target_link_libraries(${objlib} PRIVATE Blosc2::blosc2_static ${CMAKE_DL_LIBS})
# install the static library if not using the installed lib.
if(TARGET blosc2_static AND NOT Blosc2_FOUND)
install(TARGETS blosc2_static EXPORT ${objlib})
endif()
endif()

if(OPENEXR_CURLIB_PRIV_EXPORT AND BUILD_SHARED_LIBS)
target_compile_definitions(${objlib} PRIVATE ${OPENEXR_CURLIB_PRIV_EXPORT})
if(WIN32)
Expand Down
3 changes: 2 additions & 1 deletion cmake/OpenEXR.pc.in
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,13 @@ libdir=@PKG_CONFIG_INSTALL_LIBDIR@
includedir=@PKG_CONFIG_INSTALL_INCLUDEDIR@
OpenEXR_includedir=${includedir}/OpenEXR
libsuffix=@LIB_SUFFIX_DASH@
libbuildsuffix=@LIB_BUILD_SUFFIX@

Name: OpenEXR
Description: OpenEXR image library
Version: @OPENEXR_VERSION@

Libs: @exr_pthread_libs@ -L${libdir} -lOpenEXR${libsuffix} -lOpenEXRUtil${libsuffix} -lOpenEXRCore${libsuffix} -lIex${libsuffix} -lIlmThread${libsuffix}
Libs: @exr_pthread_libs@ -L${libdir} -lOpenEXR${libsuffix} -lOpenEXRUtil${libsuffix} -lOpenEXRCore${libsuffix} -lIex${libsuffix} -lIlmThread${libsuffix} -lblosc2${libbuildsuffix} -ldl
Cflags: -I${includedir} -I${OpenEXR_includedir} @exr_pthread_cflags@
Requires: Imath
Requires.private: @EXR_DEFLATE_PKGCONFIG_REQUIRES@
Expand Down
10 changes: 4 additions & 6 deletions cmake/OpenEXRConfig.cmake.in
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,10 @@

include(CMakeFindDependencyMacro)

set(openexr_needthreads @OPENEXR_ENABLE_THREADING@)
if (openexr_needthreads)
set(THREADS_PREFER_PTHREAD_FLAG ON)
find_dependency(Threads)
endif()
unset(openexr_needthreads)
# blosc2 needs threads, so we set it irrespective of OPENEXR_ENABLE_THREADING
# which enables threaded processing of requests.
set(THREADS_PREFER_PTHREAD_FLAG ON)
find_dependency(Threads)

find_dependency(Imath)

Expand Down
96 changes: 96 additions & 0 deletions cmake/OpenEXRSetup.cmake
Original file line number Diff line number Diff line change
@@ -1,6 +1,21 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright (c) Contributors to the OpenEXR Project.

function(_error_if_not_found prop var fallback)
message(STATUS "Blosc2: ${prop} ${var} '${fallback}'")
string(FIND "${var}" "-NOTFOUND" pos)
if(NOT pos EQUAL -1)
if(fallback STREQUAL "")
message(FATAL_ERROR "Blosc2: Property ${prop} not found: ${var}")
else()
string(SUBSTRING "${var}" 0 ${pos} var_name)
message(STATUS "Blosc2: Property ${prop} not found: ${var_name} falling back to '${fallback}'")
set(${var_name} "${fallback}" PARENT_SCOPE)
endif()
endif()
endfunction(_error_if_not_found)


include(GNUInstallDirs)

if(NOT "${CMAKE_PROJECT_NAME}" STREQUAL "${PROJECT_NAME}")
Expand Down Expand Up @@ -324,6 +339,87 @@ else()
endif()
endif()

#######################################
# Find or install Blosc2
#######################################

set(MINIMUM_BLOSC2_VERSION 2.11.0)
option(OPENEXR_FORCE_INTERNAL_BLOSC2 [=[Force using installed Blosc2.]=] OFF)

set(BLOSC2_REPO "https://github.com/Blosc/c-blosc2.git" CACHE STRING "Repo path for blosc2 source")
set(BLOSC2_TAG "v${MINIMUM_BLOSC2_VERSION}" CACHE STRING "Tag to use for blosc2 source repo")

# Try to find a local bloc2 install if allowed to.
if(NOT OPENEXR_FORCE_INTERNAL_BLOSC2)
message(STATUS "Blosc2: Looking for local install...")
set(CMAKE_IGNORE_PATH "${CMAKE_CURRENT_BINARY_DIR}/_deps/blosc2-src/config;${CMAKE_CURRENT_BINARY_DIR}/_deps/blosc2-build/config")
find_package(Blosc2 ${MINIMUM_BLOSC2_VERSION})
set(CMAKE_IGNORE_PATH)
endif()

if(NOT TARGET Blosc2::blosc2_static AND NOT Blosc2_FOUND)
# we didn't find a local install: let's get it from its repository.
if(OPENEXR_FORCE_INTERNAL_BLOSC2)
message(STATUS "Blosc2: forced internal, installing from ${BLOSC2_REPO} (${BLOSC2_TAG})")
else()
message(STATUS "Blosc2: no local blosc2 found, installing from ${BLOSC2_REPO} (${BLOSC2_TAG})")
endif()

# configure the blosc2 build
set(BUILD_BENCHMARKS OFF CACHE INTERNAL "no benchmarks")
set(BUILD_EXAMPLES OFF CACHE INTERNAL "no examples")
set(BUILD_FUZZERS OFF CACHE INTERNAL "no fuzzer")
set(BUILD_SHARED OFF CACHE INTERNAL "no shared library")
set(BUILD_TESTS OFF CACHE INTERNAL "no tests")

include(FetchContent)
FetchContent_Declare(Blosc2
GIT_REPOSITORY "${BLOSC2_REPO}"
GIT_TAG "${BLOSC2_TAG}"
GIT_SHALLOW ON
GIT_PROGRESS ON)

FetchContent_GetProperties(Blosc2)
if(NOT Blosc2_POPULATED)
message(STATUS "Blosc2: Downloading ${BLOSC2_TAG} from ${BLOSC2_REPO}...")
FetchContent_Populate(Blosc2)
add_subdirectory(${blosc2_SOURCE_DIR} ${blosc2_BINARY_DIR})
else()
message(STATUS "Blosc2: repo code has already been downloaded.")
endif()

# the install creates this but if we're using the library locally we
# haven't installed the header files yet, so need to extract those
# and make a variable for header only usage
if(TARGET Blosc2::blosc2_static)
message(STATUS "Blosc2: Setting up blosc directories")

get_target_property(blosc2inc Blosc2::blosc2_static INCLUDE_DIRECTORIES)
set(BLOSC2_INCLUDE_DIRS ${blosc2inc})

get_target_property(blosc2libdir Blosc2::blosc2_static BINARY_DIR)
set(BLOSC2_LIB_DIR ${blosc2libdir})

if(OPENEXR_RUN_FUZZ_TESTS)
target_compile_options(blosc2_static PUBLIC "-gdwarf-4")
endif()
endif()
else()
message(STATUS "Blosc2: Using installed Blosc2 ${Blosc2_VERSION} from ${Blosc2_DIR}")
# local build
if(TARGET Blosc2::blosc2_static)
message(STATUS "Blosc2: Setting up installed blosc directories")

get_target_property(blosc2inc Blosc2::blosc2_static INTERFACE_INCLUDE_DIRECTORIES)
_error_if_not_found("INTERFACE_INCLUDE_DIRECTORIES" ${blosc2inc} "")
set(BLOSC2_INCLUDE_DIRS ${blosc2inc})

get_target_property(blosc2libdir Blosc2::blosc2_static BINARY_DIR)
_error_if_not_found("BINARY_DIR" ${blosc2libdir} "")
set(BLOSC2_LIB_DIR ${blosc2libdir})
endif()
endif()

###########################################
# Check if we need to emulate vld1q_f32_x2
###########################################
Expand Down
4 changes: 3 additions & 1 deletion src/examples/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -24,5 +24,7 @@ cc_test(
"rgbaInterfaceTiledExamples.cpp",
"rgbaInterfaceTiledExamples.h",
],
deps = ["//:OpenEXR"],
deps = [
"//:OpenEXR",
],
)
39 changes: 32 additions & 7 deletions src/examples/deepExamples.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ readDeepScanlineFile (
// - allocate memory for the pixels
// - describe the layout of the A, and Z pixel buffers
// - read the sample counts from the file
// - allocate the memory requred to store the samples
// - allocate the memory required to store the samples
// - read the pixels from the file
//

Expand Down Expand Up @@ -148,7 +148,8 @@ writeDeepScanlineFile (

Array2D<half*>& dataA,

Array2D<unsigned int>& sampleCount)
Array2D<unsigned int>& sampleCount,
Compression compression = Compression::ZIPS_COMPRESSION)

{
//
Expand All @@ -170,7 +171,7 @@ writeDeepScanlineFile (
header.channels ().insert ("Z", Channel (FLOAT));
header.channels ().insert ("A", Channel (HALF));
header.setType (DEEPSCANLINE);
header.compression () = ZIPS_COMPRESSION;
header.compression () = compression;

DeepScanLineOutputFile file (filename, header);

Expand Down Expand Up @@ -255,8 +256,32 @@ deepExamples ()
testDataZ.resizeErase (h, w);
drawImage2 (testDataA, testDataZ, w, h);

writeDeepScanlineFile (
"test.deep.exr", window, window, dataZ, dataA, sampleCount);
readDeepScanlineFile (
"test.deep.exr", window, window, dataZ, dataA, sampleCount);
{
writeDeepScanlineFile (
"test.deep.exr",
window,
window,
dataZ,
dataA,
sampleCount,
Compression::ZSTD_COMPRESSION);
}
{
writeDeepScanlineFile (
"test.zips.exr",
window,
window,
dataZ,
dataA,
sampleCount,
Compression::ZIPS_COMPRESSION);
}
{
readDeepScanlineFile (
"test.deep.exr", window, window, dataZ, dataA, sampleCount);
}
{
readDeepScanlineFile (
"test.zips.exr", window, window, dataZ, dataA, sampleCount);
}
}
22 changes: 16 additions & 6 deletions src/examples/deepTiledExamples.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -158,11 +158,12 @@ getSampleDataForTile (

void
writeDeepTiledFile (
const char filename[],
Box2i displayWindow,
Box2i dataWindow,
int tileSizeX,
int tileSizeY)
const char filename[],
Box2i displayWindow,
Box2i dataWindow,
int tileSizeX,
int tileSizeY,
Compression compression = Compression::ZIPS_COMPRESSION)
{
//
// Write a deep image with only a A (alpha) and a Z (depth) channel,
Expand All @@ -182,7 +183,7 @@ writeDeepTiledFile (
header.channels ().insert ("Z", Channel (FLOAT));
header.channels ().insert ("A", Channel (HALF));
header.setType (DEEPTILE);
header.compression () = ZIPS_COMPRESSION;
header.compression () = compression;

header.setTileDescription (
TileDescription (tileSizeX, tileSizeY, ONE_LEVEL));
Expand Down Expand Up @@ -281,4 +282,13 @@ deepTiledExamples ()
"testTiled.deep.exr", window, window, tileSizeX, tileSizeY);
readDeepTiledFile (
"testTiled.deep.exr", window, window, dataZ, dataA, sampleCount);
writeDeepTiledFile (
"testTiled.deep.zstd.exr",
window,
window,
tileSizeX,
tileSizeY,
Compression::ZSTD_COMPRESSION);
readDeepTiledFile (
"testTiled.deep.zstd.exr", window, window, dataZ, dataA, sampleCount);
}
3 changes: 3 additions & 0 deletions src/lib/OpenEXR/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ openexr_define_library(OpenEXR
ImfTiledMisc.h
ImfZip.h
ImfZipCompressor.h
ImfZstdCompressor.h
b44ExpLogTable.h
dwaLookups.h
ImfAcesFile.cpp
Expand Down Expand Up @@ -124,6 +125,7 @@ openexr_define_library(OpenEXR
ImfWav.cpp
ImfZip.cpp
ImfZipCompressor.cpp
ImfZstdCompressor.cpp
HEADERS
ImfAcesFile.h
ImfArray.h
Expand Down Expand Up @@ -217,6 +219,7 @@ openexr_define_library(OpenEXR
ImfXdr.h
DEPENDENCIES
Imath::Imath
Blosc2::blosc2_static
OpenEXR::Config
OpenEXR::Iex
OpenEXR::IlmThread
Expand Down
3 changes: 2 additions & 1 deletion src/lib/OpenEXR/ImfCRgbaFile.h
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,8 @@ typedef struct ImfRgba ImfRgba;
#define IMF_B44A_COMPRESSION 7
#define IMF_DWAA_COMPRESSION 8
#define IMF_DWAB_COMPRESSION 9
#define IMF_NUM_COMPRESSION_METHODS 10
#define IMF_ZSTD_COMPRESSION 10
#define IMF_NUM_COMPRESSION_METHODS 11

/*
** Channels; values must be the same as in Imf::RgbaChannels.
Expand Down
Loading
Loading