YScope's Compressed Log Processor (CLP) is a tool capable of losslessly compressing text logs and searching the compressed logs without decompression. To learn more about it, you can read our paper.
You can download a release package which includes support for distributed compression and search. Or, to quickly try CLP's core compression and search, you can use a prebuilt container.
We also have guides for building the package and CLP core from source.
For some logs you can use to test CLP, check out our open-source datasets.
You can use GitHub issues to report a bug or request a feature.
Join us on Zulip to chat with developers and other community members.
CLP is currently split across a few different components in the components directory:
- clp-package-utils contains Python utilities for operating the CLP package.
- clp-py-utils contains Python utilities common to several of the other components.
- compression-job-handler contains code to submit compression jobs to a cluster.
- core contains code to compress uncompressed logs, decompress compressed logs, and search compressed logs.
- job-orchestration contains code to schedule compression jobs on the cluster.
- package-template contains the base directory structure and files of the CLP package.
The artifacts published to GitHub packages in this repo are a set of Docker container images useful for building and running CLP:
| Image name | Image contents | Link |
|---|---|---|
ghcr.io/y-scope/clp/clp-core-dependencies-x86-centos7.4:main |
The dependencies necessary to build CLP core in a Centos 7.4 x86 environment. | ↗ |
ghcr.io/y-scope/clp/clp-core-dependencies-x86-ubuntu-focal:main |
The dependencies necessary to build CLP core in an Ubuntu Focal x86 environment. | ↗ |
ghcr.io/y-scope/clp/clp-core-dependencies-x86-ubuntu-jammy:main |
The dependencies necessary to build CLP core in an Ubuntu Jammy x86 environment. | ↗ |
ghcr.io/y-scope/clp/clp-core-x86-ubuntu-focal:main |
The CLP core binaries (clg, clp, clp-s, etc.) built in an Ubuntu Focal x86 environment. |
↗ |
ghcr.io/y-scope/clp/clp-execution-x86-ubuntu-focal:main |
The dependencies necessary to run the CLP package in an x86 environment. | ↗ |
This is our open-source release which we will be constantly updating with bug fixes, features, etc. If you would like a feature or want to report a bug, please file an issue and we'll be happy to engage.
Have an issue you want to fix or a feature you'd like to implement? We'd love to see it!
Before submitting a PR, ensure you've run our linting tools and either fixed any violations or suppressed the warning. To run our linting workflows locally, you'll need Task. Alternatively, you can run the clp-lint workflow in your fork.
To perform the linting checks:
task lint-checkTo also apply any automatic fixes:
task lint-fix