README.md

CLP Core

CLP core is the low-level component that performs compression, decompression, and search.

Contents

• Requirements
• Building
  • Source Dependencies
  • Environment
    • Native Environment
    • Docker Environment
  • Build
• Running

Requirements

• We have built and tested CLP on the OSes listed below.
  • If you have trouble building for another OS, file an issue, and we may be able to help.
• A compiler that supports C++17 and std::span (e.g., gcc-10)

Building

• To build, we require some source dependencies, packages from package managers, and libraries built from source.

Source Dependencies

We use both git submodules and third-party source packages. To download all, you can run this script:

tools/scripts/deps-download/download-all.sh

This will download:

• abseil-cpp (20230802.1)
• ANTLR (v4.13.1)
• Catch2 (v2.13.7)
• date (v3.0.1)
• json (v3.10.4)
• log-surgeon (895f464)
• simdjson (v3.6.3)
• SQLite3 (v3.36.0)
• yaml-cpp (v0.7.0)

Environment

A handful of packages and libraries are required to build CLP. There are two options to use them:

• Install them on your machine and build CLP natively
• Build CLP within a prebuilt docker container that contains the libraries; However, this won't work if you need additional libraries that aren't already in the container.

Native Environment

See the relevant README for your OS:

• CentOS 7.4
• macOS 12
• Ubuntu 20.04
• Ubuntu 22.04

Want to build natively on an OS not listed here? You can file a feature request.

Docker Environment

You can use these commands to start a container in which you can build and run CLP:

# Make sure to change /path/to/clp/components/core and /path/to/my/logs below
docker run --rm -it \
  --name 'clp-build-env' \
  -u$(id -u):$(id -g) \
  -v$(readlink -f /path/to/clp/components/core):/mnt/clp \
  -v$(readlink -f /path/to/my/logs):/mnt/logs \
  ghcr.io/y-scope/clp/clp-core-dependencies-x86-ubuntu-focal:main \
  /bin/bash -l

cd /mnt/clp

Make sure to change /path/to/clp/components/core and /path/to/my/logs to the relevant paths on your machine.

Build

• Configure the cmake project:

  mkdir build
  cd build
  cmake ../

• Build:

  make -j

Running

• CLP contains three core executables: clp, clg, and clp-s.
  • clp is used for compressing and extracting unstructured (plain text) logs.
  • clg is used for performing wildcard searches on the compressed unstructured logs.
  • clp-s is used for compressing and searching semi-structured logs (e.g., JSON) with support for handling highly dynamic schemas.

See Using CLP for unstructured logs and Using CLP for semi-structured logs for usage instructions.