[JS] Cloud monitoring metrics logging

[vulus98]

Hi people, I have a question regarding logging metrics and tracing with Genkit. Namely, last month we used Genkit in production in a couple of cloud functions on GCP and noticed a big spike in billing category for Cloud Monitoring (Cloud logging costs are 0):

Since we used a default Genkit configuration with logging metrics and storing traces, like this:

flowStateStore: "firebase",
  traceStore: "firebase",
  logLevel: "error",
  enableTracingAndMetrics: false,
  telemetry: {
    instrumentation: "googleCloud",
    logger: "googleCloud",
  },

our first suspect was that the Genkit was by default logging some metrics, and that indeed seems to be the case, where, by inspecting the active billable metrics in the metrics management tab of the GCP console, we have 6 of those active related to Genkit (these are the only active billable ones for our project currently, another screenshot down there):

In order to cut the costs, we tried changing the Genkit configuration and disabling the tracing and metrics, as here:

  // flowStateStore: "firebase",
 // traceStore: "firebase",
 logLevel: "error",
 enableTracingAndMetrics: false,
 // telemetry: {
 //   instrumentation: "googleCloud",
 //   logger: "googleCloud",
 // },

Nonetheless, the metrics are still being logged when looking at metrics management tab or printing a chart in the metrics explorer, which seems confusing. Feel free to request any additional info or context which would help you to narrow down the cause of the problem. Thank you in advance!

To Reproduce
Running the Genkit plugin in a cloud function in production in GCP with specified configuration.

Expected behavior
Setting the configuration as specified should stop logging the aforementioned metrics and cut Cloud Computing costs.

[bryanatkinson]

We have confirmed that disabling the cloud plugin or disabling tracing and metrics does prevent the export of metrics data. The continued writing of this data was caused by a dangling cloud function that was not updated and continuing to receive traffic.

However the ingested bytes for those metrics is much higher than we anticipated, so we made a few adjustments to significantly reduce that:

• Removed some high-cardinality labels
• Reduced the default export frequency from 10s to 60s
• Added configuration to allow for explicit disabling of metrics and/or trace data