How to collect embeddings usage?

[onhate]

Similar to the callbacks on the BaseLanguageModel how would you collect metrics of embeddings generation?

I've done this by now, but I wonder if adding callbacks to the Embeddings API would be viable.

class ExtendedOpenAIEmbeddings extends OpenAIEmbeddings {
  constructor(settings: Partial<OpenAIEmbeddingsParams>, config: Configuration) {
    super(settings, config);

    // overwriting the private method embeddingWithRetry to add analytics metrics of usage
    const self = this as any;
    const originalEmbeddingWithRetry = self.embeddingWithRetry;
    self.embeddingWithRetry = async request => {
      // call the original method
      const response = await originalEmbeddingWithRetry.call(self, request);
      // log usage
      if (response.data?.usage) {
        const usage = response.data.usage;
        console.log(usage); // save metrics somewhere
      }
      // return as if nothing has changed
      return response;
    };
  }
}

[devstein]

@onhate I think your approach is simplest way to support this now. Here is a related issue: #586

[onhate]

I'll try to help on that, I've already checked how callbacks and callbackManager works but it's kind of messy and confusing.