Add observability

[geoand]

Also, take a look at https://smith.langchain.com/

[geoand]

• Metrics for LLM calls
• Introduce audit service concept that will be used by the logger if it exists
  ** We will need to introduce the proper context
  ** We need to make sure we don't log sensitive info
• Add Tracing
  ** We need spans per tool call in order to get insights into the entire lifecycle of the request

[geoand]

@cescoffier what do you have in mind for the audit service?

[cescoffier]

@geoand

The audit service would be an optional interface that the user can implement to store the interactions with the LLMs. It's like the access.log but for LLM - and the format should be defined by the users (as there is a good chance it needs to go in a DB).

Basically, the interface will do something like:

Audit initAudit(String systemMessage, String userMessage)

with Audit the object on which we can report the additional messages and RAG:

Audit addLLMToApplicationMessage(...)
Audit addApplicationToLLMMessage(...)
Audit addRelevantDocument(...)
void onCompletion(); // Complete the audit -> persist it, log it
void onFailure(Throwable t); // The LLM interaction failed

Each recorded step should have the message direction (LLM -> App or App -> LLM) and the timestamp.

Unknowns:

• Should audi be part of the same transaction as the LLM interaction - I would say yes. But it's rather convoluted to force that behavior in a first step.

[cescoffier]

BTW, I went with this design because the multiple message nature is tricky. Also, I would add the memory Id is available.

[geoand]

#83 is what I am starting with.

I am thinking of leaving most things unimplemented and letting the users decide what they want to do.

[geoand]

Closing as we already have this