Seeking the Ultimate Log Management Solution: Petabyte-Scale Showdown! (CLP vs Siglens vs Quickwit)

[MyraBaba]

Request

I hope i am posting the right place :)

We're on a quest to replace Splunk with an open-source log management system that can handle petabytes of data, deliver lightning-fast searches, and won't break the bank. Help us choose!
Our Wishlist 📋

Scale: Petabyte-level log management
Performance: Blazing fast search speeds
Cost: Budget-friendly
Storage: Efficient compression
Maintainability: Easy to support and develop

Contenders 🥊

1. CLP (C++)
   Pros:

Native C++ performance
Familiar technology stack for our team
Promising compression capabilities

2. Siglens (Go)
   Pros:

Appears very fast
Feature-rich

Concerns:

Potential performance overhead with Go compared to C++

3. Quickwit (Rust)
   Pros:

Modern Rust implementation
Potentially robust design

Concerns:

Limited team knowledge of Rust

🤔 Community, We Need Your Wisdom!

Which solution have you used at scale?
Performance benchmarks?
Real-world experiences with these tools?
Hidden gems we might have missed?

Bonus Points 🏆 If you can share:

Search latency comparisons
Storage efficiency metrics
Total Cost of Ownership (TCO) insights

Let's crowdsource the ultimate log management solution! Drop your knowledge bombs below! 💥
Disclaimer: Open to hearing about alternatives not mentioned here!

Possible implementation

to choose and promote CLP better and understand differentiating

[kirkrodrigues]

Hi @MyraBaba,

Thanks for starting the discussion. Here are some tidbits we can share on the CLP-side of things:

• CLP has been used in one form or another to compress and search Petabytes of log data at Uber (see Reducing Logging Cost by Two Orders of Magnitude using CLP and Modernizing Logging at Uber with CLP (Part II)).
• We do have some benchmarks available comparing CLP against other log management tools.
  • We're working on expanding this set, adding more insights, adding nightly runs, etc.
  • We'll try to add SigLens and Quickwit to the set of benchmarked tools.
  • If you have tools, datasets, or queries you'd like to see in the benchmark, let us know or feel free to contribute some!
• Although CLP hasn't reached feature parity with Splunk, we are constantly adding new and unique features like our integration of log viewing (see the new blog we just posted: A refreshed log viewer for text and JSON logs).

[MyraBaba]

Hi, Is there any online demo that we can test demo logs ?

…

On 10 Dec 2024, at 00:23, kirkrodrigues ***@***.***> wrote: Hi @MyraBaba <https://github.com/MyraBaba>, Thanks for starting the discussion. Here are some tidbits we can share on the CLP-side of things: CLP has been used in one form or another to compress and search Petabytes of log data at Uber (see Reducing Logging Cost by Two Orders of Magnitude using CLP <https://www.uber.com/blog/reducing-logging-cost-by-two-orders-of-magnitude-using-clp/> and Modernizing Logging at Uber with CLP (Part II) <https://www.uber.com/blog/modernizing-logging-with-clp-ii/>). We do have some benchmarks <https://benchmarks.yscope.com/clp/> available comparing CLP against other log management tools. We're working on expanding this set, adding more insights, adding nightly runs, etc. We'll try to add SigLens and Quickwit to the set of benchmarked tools. If you have tools, datasets, or queries you'd like to see in the benchmark, let us know or feel free to contribute some! Although CLP hasn't reached feature parity with Splunk, we are constantly adding new and unique features like our integration of log viewing (see the new blog we just posted: A refreshed log viewer for text and JSON logs <https://blog.yscope.com/a-refreshed-log-viewer-for-text-and-json-logs-85f4990ee2f5>). — Reply to this email directly, view it on GitHub <#626 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEFRZH7AAUPMQB67NK6MGSD2EYC3TAVCNFSM6AAAAABTHCTFIKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMRZGUYDONZTG4>. You are receiving this because you were mentioned.

[kirkrodrigues]

Hi MyraBaba,

You can request a free trial of our online cloud demo.

[MyraBaba]

Hi, Would you mind to to share your email to discuss business opportunities ? Best

…

On 13 Dec 2024, at 03:03, kirkrodrigues ***@***.***> wrote: Hi MyraBaba, You can request a free trial <https://docs.google.com/forms/d/e/1FAIpQLSeUvOSF-lHp386VGT30i8LfPHdbGZlj_7MN0vIAAJMpcswf_g/viewform> of our online cloud demo. — Reply to this email directly, view it on GitHub <#626 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEFRZH4ASLRKEFKIYVAVXPL2FIP5BAVCNFSM6AAAAABTHCTFIKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNBQGI2DSMRUGQ>. You are receiving this because you were mentioned.

[kirkrodrigues]

Hi MyraBaba,

It's:

[email protected]