Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(core): lock exact version of spdlog and fmtlib transitive dependency #621

Open
wants to merge 129 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
129 commits
Select commit Hold shift + click to select a range
0dc0748
Lock exact version of spdlog and its dependency fmtlib
Dec 3, 2024
266affb
Fix compilation issue on MacOS
Dec 4, 2024
0ce2faf
Fix cmake compilation issue
Dec 4, 2024
114f075
clp-package: Add Python __future__.annotations import to allow runnin…
kirkrodrigues Jun 10, 2024
2a1f85f
gh-actions: Update to latest versions of all actions. (#436)
kirkrodrigues Jun 11, 2024
619fdb0
core-clp: Add CLI command to extract a compressed file as IR. (#420)
haiqi96 Jun 12, 2024
00a7f6c
pr-template: Remove unnecessary references section and add instructio…
kirkrodrigues Jun 12, 2024
8bc5f5e
clo: Add orig_file_id and log_event_ix to search results. (#424)
haiqi96 Jun 12, 2024
8283916
lint: Apply latest version of eslint-config-yscope. (#441)
kirkrodrigues Jun 13, 2024
0af9f08
clp-s: Add archive_id to search results. (#435)
haiqi96 Jun 14, 2024
4ab24c4
clp-s: Add support for decompression in ascending timestamp order. (#…
gibber9809 Jun 14, 2024
28ac70c
Add log-viewer-webui component with boilerplate Fastify server. (#442)
kirkrodrigues Jun 14, 2024
bec6191
docs: Fix unreferenced doc. (#443)
kirkrodrigues Jun 15, 2024
f7c07fb
ffi: Extract some serialization methods for use in other files. (#444)
LinZhihao-723 Jun 16, 2024
02b648a
core: Add C++ classes for read-only memory-mapped files and UNIX file…
LinZhihao-723 Jun 17, 2024
fdda85b
package: Rename "search" to "query" in search-cluster related variabl…
haiqi96 Jun 18, 2024
4cc27e3
clo: Add CLI command to extract a compressed file as IR. (#437)
haiqi96 Jun 22, 2024
262b2d5
ffi: Add WIP Serializer class for the next IR stream format. (#446)
LinZhihao-723 Jun 22, 2024
abdbfcf
Taskfile: Replace utility tasks with ones from `yscope-dev-utils`. (#…
kirkrodrigues Jun 24, 2024
e614a48
package: Add support for different query job types: (#452)
haiqi96 Jun 24, 2024
e3eacca
clp-s: Add support for chunking output into different files during ti…
gibber9809 Jun 25, 2024
a564af2
docs: Add tooling to check and validate build: (#447)
kirkrodrigues Jun 25, 2024
d17df24
clp-s: Ensure ArchiveWriterOption is fully initialized when splitting…
gibber9809 Jun 25, 2024
242bb37
Add support for validating and escaping UTF-8 strings. (#453)
LinZhihao-723 Jun 27, 2024
589689d
clp-package: Add handling for IR extraction jobs to the query schedul…
haiqi96 Jun 28, 2024
a9c9abe
clp-s: Correctly report uncompressed size of archives during archive-…
gibber9809 Jul 3, 2024
cb0ebff
clo: Use relative path for file metadata output by an IR extraction j…
haiqi96 Jul 4, 2024
1f0b637
log-viewer-webui: Add boilerplate React client. (#468)
kirkrodrigues Jul 5, 2024
19eecef
clp-package: Deduplicate and clean up code in CLI job-launcher script…
haiqi96 Jul 6, 2024
bd846f6
log-viewer-webui: Serve client using server in production; Add npm sc…
kirkrodrigues Jul 7, 2024
2d49542
linting: Add log-viewer-webui to js-lint tasks. (#475)
kirkrodrigues Jul 8, 2024
c213a7d
clp-core: Replace calls to incomplete UTF-8 validation function with …
LinZhihao-723 Jul 11, 2024
d9bb90c
clp-package: Add log-viewer-webui as a component. (#476)
haiqi96 Jul 12, 2024
aa4cfc7
Taskfiles: Rearrange task attributes to match latest style guide. (#478)
Henry8192 Jul 15, 2024
ad41ef5
clp-package: Fix bugs introduced in #460: (#481)
haiqi96 Jul 15, 2024
3fd34cb
ffi: Add support for serializing msgpack map into KV-pair IR format. …
LinZhihao-723 Jul 17, 2024
d0db46c
core: Replace `boost-outcome` with `outcome`. (#484)
LinZhihao-723 Jul 18, 2024
54a1c21
Add initial implementation of regex_utils library containing a regex …
Bill-hbrhbr Jul 19, 2024
236ea17
clp-package: Add support for extracting files as IR streams through t…
haiqi96 Jul 19, 2024
d8df8f0
core-clp: Fix ArchiveMetadata deserialization by deserializing previo…
LinZhihao-723 Jul 19, 2024
92d78d3
log-viewer-webui: Add server implementation for submitting IR extract…
wraymo Jul 20, 2024
7700567
clp-package: Prevent query scheduler from scheduling two IR extractio…
haiqi96 Jul 23, 2024
b1a9f86
clp-s: Add option to record metadata about decompressed archive chunk…
gibber9809 Jul 23, 2024
4a117d6
regex-utils: Add support for handling escaped regex metacharacters. (…
Bill-hbrhbr Jul 24, 2024
8e1306e
clp-package: Add the log-viewer and serve it using the log-viewer-web…
Henry8192 Jul 24, 2024
b96f5a1
core-clp: Add class to encapsulate `libcurl`'s global resource manage…
LinZhihao-723 Jul 26, 2024
e9d5b2b
Add support for viewing search results in context for text logs (clp-…
junhaoliao Jul 28, 2024
fbc475e
Taskfile: Remove container OS name and version from built package nam…
kirkrodrigues Jul 28, 2024
0148b5d
webui: Optimize Meteor bundle size (fixes #498). (#499)
junhaoliao Jul 28, 2024
16ef663
clp-package: Add support for running package as root (fixes #500). (#…
haiqi96 Jul 29, 2024
1c6c925
regex-utils: Add support for translating regex character sets into wi…
Bill-hbrhbr Jul 30, 2024
ea3b93c
core-clp: Add `EncodedTextAst` class to represent parsed and encoded …
LinZhihao-723 Jul 30, 2024
cda4455
core: Add utilities for generating SHA256 hashes. (#491)
haiqi96 Jul 31, 2024
ed09024
Taskfile: Upgrade Meteor.js version to 2.16 (fixes #505). (#506)
junhaoliao Aug 1, 2024
e23f803
core: Add unit test for LogEventSerializer. (#431)
haiqi96 Aug 2, 2024
d8d21ad
ffi: Add `Value` class to represent all supported primitive values fo…
LinZhihao-723 Aug 8, 2024
2a46b42
clp-s: Report exactly where parsing error occurs when parsing JSON (f…
gibber9809 Aug 12, 2024
b7ad520
clp-core: Refactor FileReader to use RAII. (#496)
haiqi96 Aug 14, 2024
aad78ce
ffi: Add class for key-value pair log events. (#507)
LinZhihao-723 Aug 15, 2024
fef69ca
core: Add `Array` template class to abstract a fixed-size contiguous …
LinZhihao-723 Aug 18, 2024
a7a5dab
webui: Improve UI of links for viewing search results in context. (#515)
junhaoliao Aug 19, 2024
c42d1bb
core: Add signer class to generate presigned S3 URL. (#494)
haiqi96 Aug 21, 2024
8f4f2fc
core: Replace `typedef`s with type aliases. (#520)
AVMatthews Aug 21, 2024
02498db
clp-core: Add a new FileReader class that uses system call without bu…
haiqi96 Aug 22, 2024
8ec439a
core: Add support for retrieving CURL error messages, handle unexpect…
LinZhihao-723 Aug 22, 2024
b62465e
regex-utils: Use `PUBLIC` include visibility for public library heade…
LinZhihao-723 Aug 23, 2024
d7f073a
ffi: Add `Deserializer` class to deserialize log events from key-valu…
LinZhihao-723 Aug 25, 2024
2699ca9
core: Mark spdlog specialization functions as const (fixes #524). (#525)
kirkrodrigues Aug 29, 2024
bc7cf14
Refactor dependency downloads: (#508)
haiqi96 Aug 29, 2024
f6c657d
lint: Fix violations in log-viewer-webui and webui. (#529)
Henry8192 Sep 3, 2024
412e0e2
core: Link reducer-server against missing dynamic libraries; Upgrade …
kirkrodrigues Sep 5, 2024
73356b0
Rebased
kirkrodrigues Sep 6, 2024
c616405
core: Replace `std::unique_ptr<char[]>` with `Array<char>` in `Networ…
LinZhihao-723 Sep 9, 2024
20ff690
ffi: Add support for serializing a `KeyValuePairLogEvent` as a `nlohm…
LinZhihao-723 Sep 9, 2024
07b467d
log-viewer-webui: Update to latest version of yscope-log-viewer to re…
Henry8192 Sep 9, 2024
6535efe
ffi: Fix some missing errors and spelling mistakes in the KV-pair IR …
kirkrodrigues Sep 9, 2024
ce76279
docs: Fix dependency install script path and add missing requirement …
zechenericduan Sep 9, 2024
53534b9
Add OSDI'24 paper on JSON log handling to the README. (#538)
diy1 Sep 13, 2024
eb9e7ea
Linting: Upgrade to latest version of yscope-dev-utils and add C++ li…
kirkrodrigues Sep 13, 2024
45aeaaa
Temporarily lock clang-format to v18.x to resolve lint failures. (#546)
LinZhihao-723 Sep 24, 2024
7b10f41
ffi: Add `IrUnitHandlerInterface` to perform user-defined handling fo…
LinZhihao-723 Sep 28, 2024
6385cb6
ffi: Extract IR unit deserialization methods from `Deserializer` into…
LinZhihao-723 Sep 30, 2024
8b41436
gh-actions: Remove pre-installed Go binaries that conflict with brew'…
LinZhihao-723 Oct 4, 2024
b499711
core: Extract `TransactionManager` from `clp::ffi::ir_stream::Deseria…
LinZhihao-723 Oct 4, 2024
071edce
clp-s: Add support for projecting of a subset of columns during searc…
gibber9809 Oct 7, 2024
b6535c1
ffi: Redesign `Deserializer` API to deserialize key-value pair IR str…
LinZhihao-723 Oct 7, 2024
7f822cb
Temporarily lock Sphinx to v8.0.2 to resolve docs build failure. (#555)
LinZhihao-723 Oct 11, 2024
2c84316
ffi: Move `SchemaTreeNode` into `SchemaTree`; Use an `optional` for t…
LinZhihao-723 Oct 12, 2024
3e8335b
ffi: Add support for serializing/deserializing auto-generated and use…
LinZhihao-723 Oct 18, 2024
cdea01f
clp: Add missing C++ standard library includes in IR parsing files. (…
LinZhihao-723 Oct 22, 2024
7159e21
log-viewer-webui: Update `yscope-log-viewer` to the latest version (w…
junhaoliao Oct 24, 2024
5ece291
package: Upgrade dependencies to resolve security issues. (#536)
Henry8192 Oct 24, 2024
6f7d65f
clp-s: Implement table packing (#466)
gibber9809 Oct 25, 2024
9610ab5
log-viewer-webui: Update `yscope-log-viewer` to the latest version. (…
kirkrodrigues Oct 28, 2024
2feb739
ci: Switch GitHub macOS build workflow to use macos-13 (x86) and maco…
kirkrodrigues Oct 29, 2024
58f98ad
core: Add support for user-defined HTTP headers in `NetworkReader`. (…
anlowee Oct 31, 2024
76c9bc9
chore: Update to the latest version of yscope-dev-utils. (#574)
junhaoliao Nov 7, 2024
c14c4ca
build(core): Upgrade msgpack to v7.0.0. (#575)
LinZhihao-723 Nov 7, 2024
e88e72a
feat(ffi): Update IR stream protocol version handling in preparation …
LinZhihao-723 Nov 7, 2024
e29e6df
fix(taskfiles): Trim trailing slash from URL prefix in `download-and-…
kirkrodrigues Nov 8, 2024
e8a9470
fix(ffi): Correct `clp::ffi::ir_stream::Deserializer::deserialize_nex…
LinZhihao-723 Nov 8, 2024
140f526
fix(taskfiles): Update `yscope-log-viewer` sources in `log-viewer-web…
junhaoliao Nov 8, 2024
1e53f2c
fix(cmake): Add Homebrew path detection for `mariadb-connector-c` to …
LinZhihao-723 Nov 10, 2024
7a1c842
refactor(ffi): Make `get_schema_subtree_bitmap` a public method of `K…
LinZhihao-723 Nov 10, 2024
fb70484
ci: Schedule GitHub workflows to daily run to detect failures due to …
LinZhihao-723 Nov 10, 2024
ee8c8ad
docs: Update the required version of task. (#567)
haiqi96 Nov 11, 2024
e50c423
docs(ffi): Update `ffi::ir_stream::Serializer`'s doc string to remove…
LinZhihao-723 Nov 12, 2024
4543d61
ci: Add GH workflow to validate PR titles follow Conventional Commits…
LinZhihao-723 Nov 12, 2024
2a81913
feat: Add support for escaping characters in KQL key names. (#560)
gibber9809 Nov 13, 2024
62538f3
build(core-clp): Remove `boost::iostream` dependency. (#450)
LinZhihao-723 Nov 15, 2024
a48b066
test(core): Update `network_reader_with_valid_http_header_kv_pairs` t…
LinZhihao-723 Nov 15, 2024
7054958
feat(clp-package): Add support for extracting JSON streams from archi…
haiqi96 Nov 18, 2024
afdb394
feat(clp-s): Record log-order at compression time. (#584)
gibber9809 Nov 19, 2024
e095213
feat(clp-s): Chunk output by size (in bytes) during ordered decompres…
gibber9809 Nov 20, 2024
84f7366
feat(webui): Support viewing search results in context for JSON logs …
haiqi96 Nov 21, 2024
44604c1
fix(gh-macos-build): Check if pkg-config is installed before trying t…
kirkrodrigues Nov 22, 2024
a402054
refactor(core): Fix clang-tidy warnings in the streaming compressor i…
Bill-hbrhbr Nov 23, 2024
4d1e9e6
feat(clp-package): Add support for deleting archives that are exclusi…
haiqi96 Nov 27, 2024
1da88c6
feat(clp-s): Add the write path for single-file archives. (#563)
wraymo Nov 27, 2024
7632955
test: Allow multiple trials when unittesting http headers (#613)
anlowee Nov 28, 2024
a1fd272
chore(log-viewer-webui): Update `yscope-log-viewer` to the latest ver…
junhaoliao Nov 29, 2024
95d4425
test(clp-s): Add end-to-end test case for compression and extraction.…
AVMatthews Dec 2, 2024
3329aae
docs(clp-json): Update list of characters that requires escaping in q…
gibber9809 Dec 2, 2024
f5f1613
feat(core): Add `ErrorCode` template to standardize conversion of use…
LinZhihao-723 Dec 2, 2024
52a16f8
Fix compilation issue on MacOS
Dec 4, 2024
7a08ac8
Rebased and addressed code review comments
Dec 4, 2024
765b19d
Merge branch 'y-scope:main' into main-exact-lib-version
jackluo923 Dec 4, 2024
ae54f7b
Print the actual fmt version found rather than the minimum version.
Dec 4, 2024
3dc9eb7
Added validation of upperbound fmtlib version check.
Dec 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
test(clp-s): Add end-to-end test case for compression and extraction. (
  • Loading branch information
AVMatthews authored and Jack Luo committed Dec 4, 2024
commit 95d442535eb4e96ddc080e56cc9e82f1f11f1d0f
54 changes: 53 additions & 1 deletion components/core/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -241,6 +241,42 @@ add_subdirectory(src/clp_s)
add_subdirectory(src/reducer)

set(SOURCE_FILES_clp_s_unitTest
src/clp_s/ArchiveReader.cpp
src/clp_s/ArchiveReader.hpp
src/clp_s/ArchiveWriter.cpp
src/clp_s/ArchiveWriter.hpp
src/clp_s/ColumnReader.cpp
src/clp_s/ColumnReader.hpp
src/clp_s/ColumnWriter.cpp
src/clp_s/ColumnWriter.hpp
src/clp_s/DictionaryEntry.cpp
src/clp_s/DictionaryEntry.hpp
src/clp_s/DictionaryWriter.cpp
src/clp_s/DictionaryWriter.hpp
src/clp_s/FileReader.cpp
src/clp_s/FileReader.hpp
src/clp_s/FileWriter.cpp
src/clp_s/FileWriter.hpp
src/clp_s/JsonConstructor.cpp
src/clp_s/JsonConstructor.hpp
src/clp_s/JsonFileIterator.cpp
src/clp_s/JsonFileIterator.hpp
src/clp_s/JsonParser.cpp
src/clp_s/JsonParser.hpp
src/clp_s/PackedStreamReader.cpp
src/clp_s/PackedStreamReader.hpp
src/clp_s/ReaderUtils.cpp
src/clp_s/ReaderUtils.hpp
src/clp_s/Schema.cpp
src/clp_s/Schema.hpp
src/clp_s/SchemaMap.cpp
src/clp_s/SchemaMap.hpp
src/clp_s/SchemaReader.cpp
src/clp_s/SchemaReader.hpp
src/clp_s/SchemaTree.cpp
src/clp_s/SchemaTree.hpp
src/clp_s/SchemaWriter.cpp
src/clp_s/SchemaWriter.hpp
src/clp_s/search/AndExpr.cpp
src/clp_s/search/AndExpr.hpp
src/clp_s/search/BooleanLiteral.cpp
@@ -273,11 +309,24 @@ set(SOURCE_FILES_clp_s_unitTest
src/clp_s/search/StringLiteral.hpp
src/clp_s/search/Transformation.hpp
src/clp_s/search/Value.hpp
src/clp_s/SchemaTree.hpp
src/clp_s/TimestampDictionaryReader.cpp
src/clp_s/TimestampDictionaryReader.hpp
src/clp_s/TimestampDictionaryWriter.cpp
src/clp_s/TimestampDictionaryWriter.hpp
src/clp_s/TimestampEntry.cpp
src/clp_s/TimestampEntry.hpp
src/clp_s/TimestampPattern.cpp
src/clp_s/TimestampPattern.hpp
src/clp_s/Utils.cpp
src/clp_s/Utils.hpp
src/clp_s/VariableDecoder.cpp
src/clp_s/VariableDecoder.hpp
src/clp_s/VariableEncoder.cpp
src/clp_s/VariableEncoder.hpp
src/clp_s/ZstdCompressor.cpp
src/clp_s/ZstdCompressor.hpp
src/clp_s/ZstdDecompressor.cpp
src/clp_s/ZstdDecompressor.hpp
)

set(SOURCE_FILES_unitTest
@@ -499,6 +548,7 @@ set(SOURCE_FILES_unitTest
tests/LogSuppressor.hpp
tests/test-Array.cpp
tests/test-BufferedFileReader.cpp
tests/test-clp_s-end_to_end.cpp
tests/test-EncodedVariableInterpreter.cpp
tests/test-encoding_methods.cpp
tests/test-ffi_IrUnitHandlerInterface.cpp
@@ -542,6 +592,8 @@ target_link_libraries(unitTest
log_surgeon::log_surgeon
LibArchive::LibArchive
MariaDBClient::MariaDBClient
${MONGOCXX_TARGET}
simdjson
spdlog::spdlog
OpenSSL::Crypto
${sqlite_LIBRARY_DEPENDENCIES}
158 changes: 158 additions & 0 deletions components/core/tests/test-clp_s-end_to_end.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
#include <sys/wait.h>

#include <cstdlib>
#include <filesystem>
#include <string>
#include <string_view>
#include <vector>

#include <Catch2/single_include/catch2/catch.hpp>
#include <fmt/format.h>

#include "../src/clp_s/JsonConstructor.hpp"
#include "../src/clp_s/JsonParser.hpp"

constexpr std::string_view cTestEndToEndArchiveDirectory{"test-end-to-end-archive"};
constexpr std::string_view cTestEndToEndOutputDirectory{"test-end-to-end-out"};
constexpr std::string_view cTestEndToEndOutputSortedJson{"test-end-to-end_sorted.jsonl"};
constexpr std::string_view cTestEndToEndInputFileDirectory{"test_log_files"};
constexpr std::string_view cTestEndToEndInputFile{"test_no_floats_sorted.jsonl"};

namespace {
/**
* A class that deletes the directories and files created by test cases, both before and after each
* test case where the class is instantiated.
*/
class TestOutputCleaner {
public:
TestOutputCleaner() { delete_files(); }

~TestOutputCleaner() { delete_files(); }

// Delete copy & move constructors and assignment operators
TestOutputCleaner(TestOutputCleaner const&) = delete;
TestOutputCleaner(TestOutputCleaner&&) = delete;
auto operator=(TestOutputCleaner const&) -> TestOutputCleaner& = delete;
auto operator=(TestOutputCleaner&&) -> TestOutputCleaner& = delete;

private:
static void delete_files() {
std::filesystem::remove_all(cTestEndToEndArchiveDirectory);
std::filesystem::remove_all(cTestEndToEndOutputDirectory);
std::filesystem::remove(cTestEndToEndOutputSortedJson);
}
};

auto get_test_input_path_relative_to_tests_dir() -> std::filesystem::path;
auto get_test_input_local_path() -> std::string;
void compress(bool structurize_arrays);
auto extract() -> std::filesystem::path;
void compare(std::filesystem::path const& extracted_json_path);

auto get_test_input_path_relative_to_tests_dir() -> std::filesystem::path {
return std::filesystem::path{cTestEndToEndInputFileDirectory} / cTestEndToEndInputFile;
}

auto get_test_input_local_path() -> std::string {
std::filesystem::path const current_file_path{__FILE__};
auto const tests_dir{current_file_path.parent_path()};
return (tests_dir / get_test_input_path_relative_to_tests_dir()).string();
}

void compress(bool structurize_arrays) {
constexpr auto cDefaultTargetEncodedSize = 8ULL * 1024 * 1024 * 1024; // 8 GiB
constexpr auto cDefaultMaxDocumentSize = 512ULL * 1024 * 1024; // 512 MiB
constexpr auto cDefaultMinTableSize = 1ULL * 1024 * 1024; // 1 MiB
constexpr auto cDefaultCompressionLevel = 3;
constexpr auto cDefaultPrintArchiveStats = false;

std::filesystem::create_directory(cTestEndToEndArchiveDirectory);
REQUIRE((std::filesystem::is_directory(cTestEndToEndArchiveDirectory)));

clp_s::JsonParserOption parser_option{};
parser_option.file_paths.push_back(get_test_input_local_path());
parser_option.archives_dir = cTestEndToEndArchiveDirectory;
parser_option.target_encoded_size = cDefaultTargetEncodedSize;
parser_option.max_document_size = cDefaultMaxDocumentSize;
parser_option.min_table_size = cDefaultMinTableSize;
parser_option.compression_level = cDefaultCompressionLevel;
parser_option.print_archive_stats = cDefaultPrintArchiveStats;
parser_option.structurize_arrays = structurize_arrays;

clp_s::JsonParser parser{parser_option};
REQUIRE(parser.parse());
parser.store();

REQUIRE((false == std::filesystem::is_empty(cTestEndToEndArchiveDirectory)));
}

auto extract() -> std::filesystem::path {
constexpr auto cDefaultOrdered = false;
constexpr auto cDefaultTargetOrderedChunkSize = 0;

std::filesystem::create_directory(cTestEndToEndOutputDirectory);
REQUIRE(std::filesystem::is_directory(cTestEndToEndOutputDirectory));

clp_s::JsonConstructorOption constructor_option{};
constructor_option.archives_dir = cTestEndToEndArchiveDirectory;
constructor_option.output_dir = cTestEndToEndOutputDirectory;
constructor_option.ordered = cDefaultOrdered;
constructor_option.target_ordered_chunk_size = cDefaultTargetOrderedChunkSize;
for (auto const& entry : std::filesystem::directory_iterator(constructor_option.archives_dir)) {
if (false == entry.is_directory()) {
// Skip non-directories
continue;
}

constructor_option.archive_id = entry.path().filename();
clp_s::JsonConstructor constructor{constructor_option};
constructor.store();
}
std::filesystem::path extracted_json_path{cTestEndToEndOutputDirectory};
extracted_json_path /= "original";
REQUIRE(std::filesystem::exists(extracted_json_path));

return extracted_json_path;
}

// Silence the checks below since our use of `std::system` is safe in the context of testing.
// NOLINTBEGIN(cert-env33-c,concurrency-mt-unsafe)
void compare(std::filesystem::path const& extracted_json_path) {
int result{std::system("command -v jq >/dev/null 2>&1")};
REQUIRE((0 == result));
auto command = fmt::format(
"jq --sort-keys --compact-output '.' {} | sort > {}",
extracted_json_path.string(),
cTestEndToEndOutputSortedJson
);
result = std::system(command.c_str());
REQUIRE((0 == result));

REQUIRE((false == std::filesystem::is_empty(cTestEndToEndOutputSortedJson)));

result = std::system("command -v diff >/dev/null 2>&1");
REQUIRE((0 == result));
command = fmt::format(
"diff --unified {} {} > /dev/null",
cTestEndToEndOutputSortedJson,
get_test_input_local_path()
);
result = std::system(command.c_str());
REQUIRE((true == WIFEXITED(result)));
REQUIRE((0 == WEXITSTATUS(result)));
}

// NOLINTEND(cert-env33-c,concurrency-mt-unsafe)
} // namespace

TEST_CASE("clp-s-compress-extract-no-floats", "[clp-s][end-to-end]") {
auto structurize_arrays = GENERATE(true, false);

TestOutputCleaner const test_cleanup;

compress(structurize_arrays);

auto extracted_json_path = extract();

compare(extracted_json_path);
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{"clp_string":"uid=0, CPU usage:99.99%, \"user_name\"=YScope","empty_array":[],"empty_object":{},"false":false,"int16_max":32767,"int16_min":-32768,"int32_max":2147483647,"int32_min":-2147483648,"int64_max_jq_losslessly_represents":9824299763229016,"int64_min_jq_losslessly_represents":-9007199254740992,"int8_max":127,"int8_min":-128,"null":null,"string":"short_string","true":true}
{"clp_string":"uid=0, CPU usage:99.99%, \"user_name\"=YScope","empty_array":[],"false":false,"int16_max":32767,"int16_min":-32768,"int32_max":2147483647,"int32_min":-2147483648,"int64_max_jq_losslessly_represents":9824299763229016,"int64_min_jq_losslessly_represents":-9007199254740992,"int8_max":127,"int8_min":-128,"nonempty_object":{"clp_string":"uid=0, CPU usage:99.99%, \"user_name\"=YScope","empty_array":[],"empty_object":{},"false":false,"int16_max":32767,"int16_min":-32768,"int32_max":2147483647,"int32_min":-2147483648,"int64_max_jq_losslessly_represents":9824299763229016,"int64_min_jq_losslessly_represents":-9007199254740992,"int8_max":127,"int8_min":-128,"null":null,"string":"short_string","true":true},"null":null,"string":"short_string","true":true}
{"clp_string":"uid=0, CPU usage:99.99%, \"user_name\"=YScope","empty_array":[],"false":false,"int16_max":32767,"int16_min":-32768,"int32_max":2147483647,"int32_min":-2147483648,"int64_max_jq_losslessly_represents":9824299763229016,"int64_min_jq_losslessly_represents":-9007199254740992,"int8_max":127,"int8_min":-128,"nonempty_object":{"clp_string":"uid=0, CPU usage:99.99%, \"user_name\"=YScope","empty_array":[],"false":false,"int16_max":32767,"int16_min":-32768,"int32_max":2147483647,"int32_min":-2147483648,"int64_max_jq_losslessly_represents":9824299763229016,"int64_min_jq_losslessly_represents":-9007199254740992,"int8_max":127,"int8_min":-128,"non_empty_object2":{"clp_string":"uid=0, CPU usage:99.99%, \"user_name\"=YScope","empty_array":[],"empty_object":{},"false":false,"int16_max":32767,"int16_min":-32768,"int32_max":2147483647,"int32_min":-2147483648,"int64_max_jq_losslessly_represents":9824299763229016,"int64_min_jq_losslessly_represents":-9007199254740992,"int8_max":127,"int8_min":-128,"null":null,"string":"short_string","true":true},"null":null,"string":"short_string","true":true},"null":null,"string":"short_string","true":true}
{"clp_string":"uid=0, CPU usage:99.99%, \"user_name\"=YScope","empty_object":{},"false":false,"int16_max":32767,"int16_min":-32768,"int32_max":2147483647,"int32_min":-2147483648,"int64_max_jq_losslessly_represents":9824299763229016,"int64_min_jq_losslessly_represents":-9007199254740992,"int8_max":127,"int8_min":-128,"nonempty_array":[1,2,3,4,5],"null":null,"string":"short_string","true":true}
Original file line number Diff line number Diff line change
@@ -8,9 +8,11 @@ set -u

dnf install -y \
cmake \
diffutils \
gcc-c++ \
git \
java-11-openjdk \
jq \
libarchive-devel \
libcurl-devel \
libzstd-devel \
Original file line number Diff line number Diff line change
@@ -17,6 +17,7 @@ DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \
gcc \
gcc-10 \
git \
jq \
libcurl4 \
libcurl4-openssl-dev \
libmariadb-dev \
Original file line number Diff line number Diff line change
@@ -14,6 +14,7 @@ DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \
curl \
build-essential \
git \
jq \
libboost-filesystem-dev \
libboost-iostreams-dev \
libboost-program-options-dev \