Observability for streaming models

[geoand]

Discussed in #1080

^{Originally posted by cmasantos November 14, 2024}
Hi, I've notice that "normal" chat model produce a set of gen_ai* metrics. But I've been unable to use them with streaming models. Also when we use a combination of Mutiny StreamingResponseHandler I'm almost sure that it can't be done.

My question is, how can I add metrics to a streaming model even if need to instantiate them? I've found the MetricsChatModelListener class but I can't find a way to put it together with the streaming model interface.

We should add observability for StreamingChatLanguageModel as well

[geoand]

@cmasantos would you be willing to contribute this feature?

[cmasantos]

@cmasantos would you be willing to contribute this feature?

Yes, I'd like to contribute. I'll give it a try, but it will take me a while because I have no idea how extensions development work. I've got to learn that first.

[geoand]

Great!

In this case you probably don't need to learn too many details as the changes almost exactly match those that were made for ChatLanguageModel - I can certainly point you in the right direction.