(+semver: feature) Add raw topic message type

[annymsMthd]

Signed-off-by: Joshua Benjamin [email protected]

[annymsMthd]

Wanted to get this out there to get eyes on it early. I had this change local for a bit and had to do some merge craziness so I wanted to make sure the previous commit in master is gtg.

[cuevaskoch]

I'm a big fan of the snapshots for testing! Also this implementation for a raw topic type looks good to me in general, but I plan to review more carefully later. May I ask what your use case is? Message schemas are a pretty core part of the kafmesh conceptual framework IMO.

[annymsMthd]

Sure I can expand on that. We have open telemetry collectors running in all of of our clusters. We have those shipping the spans to kafka. We use the raw message type to process those spans and everything after that gets a schema. Basically ingesting data into the mesh that we don't have full control over.
#34

[annymsMthd]

name: tracing
description: The tracing handles trace processing.

processors:
  - name: ingresser
    definition: reads records from the open telemetry topic
    inputs:
      - message: opentelemetry
        topic: traces
        type: raw
    outputs:
      - message: serviceName.resourceSpans

  - name: logger
    definition: logs traceIds for searching
    inputs:
      - message: serviceName.resourceSpans
    outputs:
      - message: serviceName.spanLog

sinks:
  - name: tempo sink
    description: sinks spans into grafana tempo
    message: serviceName.resourceSpans

  - name: loki sink
    description: sinks spans logs into loki for queries
    message: serviceName.spanLog

Example of our configuration

[cuevaskoch]

👍 lgtm, thanks!