Skip to content
/ clp Public

Compressed Log Processor (CLP) is a free log management tool capable of compressing logs and searching the compressed logs without decompression.

License

Notifications You must be signed in to change notification settings

y-scope/clp

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Folders and files

NameName
Last commit message
Last commit date
Jan 11, 2024
Jan 22, 2024
Dec 24, 2023
Jan 20, 2024
Jan 12, 2024
Jan 10, 2024
Jan 9, 2024
Jan 10, 2024
Oct 1, 2021
Jan 10, 2024
Jan 12, 2024
Jan 10, 2024
Jan 6, 2024

Repository files navigation

CLP

Open bug reports Open feature requests CLP on Zulip

YScope's Compressed Log Processor (CLP) is a tool capable of losslessly compressing text logs and searching the compressed logs without decompression. To learn more about it, you can read our paper.

Getting Started

You can download a release package which includes support for distributed compression and search. Or, to quickly try CLP's core compression and search, you can use a prebuilt container.

We also have guides for building the package and CLP core from source.

For some logs you can use to test CLP, check out our open-source datasets.

Providing Feedback

You can use GitHub issues to report a bug or request a feature.

Join us on Zulip to chat with developers and other community members.

Project Structure

CLP is currently split across a few different components in the components directory:

  • clp-package-utils contains Python utilities for operating the CLP package.
  • clp-py-utils contains Python utilities common to several of the other components.
  • compression-job-handler contains code to submit compression jobs to a cluster.
  • core contains code to compress uncompressed logs, decompress compressed logs, and search compressed logs.
  • job-orchestration contains code to schedule compression jobs on the cluster.
  • package-template contains the base directory structure and files of the CLP package.

GitHub Packages

The artifacts published to GitHub packages in this repo are a set of Docker container images useful for building and running CLP:

Image name Image contents Link
ghcr.io/y-scope/clp/clp-core-dependencies-x86-centos7.4:main The dependencies necessary to build CLP core in a Centos 7.4 x86 environment.
ghcr.io/y-scope/clp/clp-core-dependencies-x86-ubuntu-focal:main The dependencies necessary to build CLP core in an Ubuntu Focal x86 environment.
ghcr.io/y-scope/clp/clp-core-dependencies-x86-ubuntu-jammy:main The dependencies necessary to build CLP core in an Ubuntu Jammy x86 environment.
ghcr.io/y-scope/clp/clp-core-x86-ubuntu-focal:main The CLP core binaries (clg, clp, clp-s, etc.) built in an Ubuntu Focal x86 environment.
ghcr.io/y-scope/clp/clp-execution-x86-ubuntu-focal:main The dependencies necessary to run the CLP package in an x86 environment.

Next Steps

This is our open-source release which we will be constantly updating with bug fixes, features, etc. If you would like a feature or want to report a bug, please file an issue and we'll be happy to engage.

Contributing

Have an issue you want to fix or a feature you'd like to implement? We'd love to see it!

Linting

Before submitting a PR, ensure you've run our linting tools and either fixed any violations or suppressed the warning. To run our linting workflows locally, you'll need Task. Alternatively, you can run the clp-lint workflow in your fork.

To perform the linting checks:

task lint-check

To also apply any automatic fixes:

task lint-fix