Skip to content

clp-s: Make error handling for invalid JSON during ingestion consistent regardless of underlying error. #589

Open
@gibber9809

Description

@gibber9809

Request

Currently depending on what kind of invalid JSON record it encounters clp-s will either report a failure directly before the invalid message, or before a (large) batch containing the invalid message. The latter case makes it difficult to find the invalid JSON and solve the issue.

Possible implementation

Instead of feeding batches of raw log data to simdjson feed simdjson one log message at a time. We can do this by using something like std::getline (assuming the underlying content is ndjson), or by using some of the same simd tricks used by simdjson.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions