IPFIX fields missing for newer netobserv features (Connection RTT, DNS decode etc)

[rupertgregoryibm]

Hello team,

As the main agent has been leaping forward, it seems the IPFIX export feature of FLP needs an update to support export of these fields -

DnsFlags
DnsFlagsResponseCode
DnsId
DnsLatencyMs
RTT
DstK8S_HostIP
DstK8S_HostName
DstK8S_Name
DstK8S_Namespace
DstK8S_OwnerName
DstK8S_OwnerType,
DstK8S_Type

....there may be (read: probably are) more fields, I've just done brief scan of various MRs over the past few months.

Please can we get the IPFIX export updated. I'm happy to allocate new IPFIX entries if someone can help or point me to the complete schema for all the possible fields and also what each one means.

Many thanks. R.

[jotak]

Some of these fields are indeed probably missing but some of them are not expected: all the "DstK8S..." stuff comes from kube enrichment stage that happens in flowlogs-pipeline ie downstream of the agent.

@msherif1234 we should look for a way to assert that all exporters are always on par with their exported fields ... Like in my current PR here I'm adding assertions that the grpc or direct-flp export produce the same result; maybe we can try to generalize to include ipfix too

[rupertgregoryibm]

Thanks for the quick response @jotak - yes, should have been clearer, theres some upstream fields from the eBPF agent and some FLP enriched fields - all should be consumable irrespective of the export method.. which you've highlighted 👍

I have also let @dushyantbehl know as well as some of this he worked on.

[jotak]

Oh I didn't realize you posted on flowlogs-pipeline repo I thought it was the ebpf agent repo ... it is a bit confusing because both the agent and FLP have the ability to export as ipfix ; so you're using IPFIX export from FLP and not from the Agent, so indeed you should get the kube enriched fields.
On the new fields such as DNS / RTT etc. I guess yes they need to be added somewhere on this file https://github.com/netobserv/flowlogs-pipeline/blob/main/pkg/pipeline/write/write_ipfix.go

[rupertgregoryibm]

@jotak is there a master field schema anywhere in the ebpf agent? I assume it would be buried in a protobuf def somewhere?

[jotak]

@rupertgregoryibm yes indeed, here's the protobuf defintion https://github.com/netobserv/netobserv-ebpf-agent/blob/main/proto/flow.proto
And here's where it's converted in FLP format: https://github.com/netobserv/netobserv-ebpf-agent/blob/main/pkg/decode/decode_protobuf.go#L44