Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

clp-s: Make error handling for invalid JSON during ingestion consistent regardless of underlying error. #589

Open
gibber9809 opened this issue Nov 13, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@gibber9809
Copy link
Contributor

Request

Currently depending on what kind of invalid JSON record it encounters clp-s will either report a failure directly before the invalid message, or before a (large) batch containing the invalid message. The latter case makes it difficult to find the invalid JSON and solve the issue.

Possible implementation

Instead of feeding batches of raw log data to simdjson feed simdjson one log message at a time. We can do this by using something like std::getline (assuming the underlying content is ndjson), or by using some of the same simd tricks used by simdjson.

@gibber9809 gibber9809 added the enhancement New feature or request label Nov 13, 2024
@gibber9809 gibber9809 self-assigned this Nov 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant