Unsupported data type when parsing optional UUID field in JSON logs

[sfriedowitz]

/about output:

{
  "version": "1.6.3",
  ...
}

Description:

We have a field uuid that shows up in our JSON logs, but may not be present for every log event. When it is not present, the logger will output a null in place of the UUID string. We get this error in our log events which are ingested by FluentBit:

2024-12-19T21:14:42.637482Z ERROR parseable::event::format::json: Unsupported datatype Null, value String("4327e7f0-be4e-11ef-864f-db81a31ad329")                                          │
│ thread 'actix-rt|system:0|arbiter:2' panicked at src/event/format/json.rs:228:13:                                                                                                          │
│ internal error: entered unreachable code

Clearly this is an issue with the inferred schema not being able to handle the optional string values for this field. Should we be doing something different, or is this a bug?

[nitisht]

We might have to look at how dynamic schema behaves - but in this case if you know the schema you're going to ingest, it might be better to use static streams: https://www.parseable.com/docs/server/concepts/ingestion#static

[sfriedowitz]

is there a way to specify an optional field that can be Null or string? From the docs, it looks like theres no notion of an "option" type for a static schema

[sfriedowitz]

I've also tried changing our log formatter so that null fields are removed from the log record, and am getting new errors in the ingestion pod after purging our old log records from S3

│ thread 'main' panicked at src/utils/arrow/batch_adapter.rs:47:47:                                                                                                                          │
│ called `Result::unwrap()` on an `Err` value: InvalidArgumentError("column types must match schema types, expected Utf8 but found Null at column index 3")                                  │
│ note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace                                                                                                              │
│ 2024-12-20T18:15:06.000019Z ERROR parseable::handlers::http::modal::server: Error joining remote_sync_handler: JoinError::Panic(Id(58), ...)