Elasticsearch Data source Plugin error after provisioning

[jcapote22]

What happened?

``When creating datasources in Grafana via Kubernetes resources, each datasource appears correctly in the Grafana UI. However, accessing any newly created datasource shows the following error:

"Query error: An error occurred within the plugin"

These datasources, managed via ArgoCD, are created using Kubernetes resources, allowing configuration as code and ensuring consistency across deployments. In this setup, we manage over 100 datasources, so the current behavior requiring manual activation is particularly challenging.

Despite this error, the datasource successfully connects after manually clicking the "Save & Test" button in the UI. This manual intervention is problematic when deploying multiple datasources, as it requires opening each datasource individually to connect it, which is inefficient.

same datasource but you give it save and test and it works

no pod restarts or log errors

datasource yaml example

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: elasticsearch-peru
  namespace: grafana-operator-system
spec:
  valuesFrom:
    - targetPath: "basicAuthUser"
      valueFrom:
        secretKeyRef:
          name: "extra-credentials"
          key: "user"
    - targetPath: "secureJsonData.basicAuthPassword"
      valueFrom:
        secretKeyRef:
          name: "extra-credentials"
          key: "pass"
  instanceSelector:
    matchLabels:
      grafana: default
  datasource:
    access: proxy
    jsonData:
      timeInterval: 5s
      tlsSkipVerify: false
      index: "banco-pe-omni*"
    secureJsonData:
      "basicAuthPassword": ${pass}
    name: ElasticBFPE
    type: elasticsearch
    url: https://internal url
    basicAuth: true
    basicAuthUser: ${user}

What did you expect to happen?

I expected Grafana to automatically validate and establish connections for each datasource upon creation, without needing to manually interact with the "Save & Test" button.

Did this work before?

This is a new setup, so it hasn't worked correctly with the current versions. I haven’t encountered this issue in previous versions, but the environment has changed recently.

How do we reproduce it?

1- Deploy Grafana with Grafana Operator and Grafana version 11.2.2.
2- Create a datasource using a Kubernetes resource.
3- Access the datasource in the Grafana UI.
4- Observe the initial error "Query error: An error occurred within the plugin".
5- Manually click "Save & Test" in the UI to establish the connection.

Is the bug inside a dashboard panel?

No, the issue occurs within the datasource configuration UI and not within a dashboard panel.

Environment (with versions)?

Grafana: 11.2.2
OS: linux (kubernetes)
Browser: chrome Versión 129.0

Grafana platform?

Kubernetes

Datasource(s)?

Elasticsearch v 7.17.19

[idastambuk]

Hi @jcapote22, if you have access to backend logs, are there any error messages logged there that appear when this happens?
Also, what is the version of Grafana Operator you're using? Thanks!

[jcapote22]

We've gathered more information to help diagnose this issue:

1. Scope of the Issue: The problem occurs exclusively with the Elasticsearch datasource. Other datasources, such as Instana and Prometheus, are unaffected and connect automatically without needing the "Save & Test" workaround.

2. Elasticsearch Deployment: Our Elasticsearch instance is running within a Kubernetes cluster.

3. Grafana User Permissions: The Grafana user interacting with the Elasticsearch datasource has been granted the roles: monitor/main, monitor/remote, and info. These permissions should be sufficient for typical query operations, so we’re unsure why the initial error occurs only for Elasticsearch.

Error Logs:

Before "Save & Test": Attaching logs showing the initial error message ("Query error: An error occurred within the plugin") encountered upon accessing the datasource.

logger=tsdb.elasticsearch endpoint=queryData pluginId=elasticsearch dsName=ElasticBFPE dsUID=b80703e4-c812-46a2-8b29-f13654b90b9a uname="juan francisco capote medina" fromAlert=false t=2024-11-08T14:11:53.32817054Z level=error msg="Failed to get data source info" error="timeField cannot be cast to string"
logger=context userId=2 orgId=1 uname="juan francisco capote medina" t=2024-11-08T14:11:53.32823614Z level=error msg=InternalError error="[plugin.downstreamError] client: failed to query data: timeField cannot be cast to string"

![Image](https://github.com/user-attachments/assets/9386b429-4962-499d-a33b-5fca82406c2f)

After "Save & Test": Attaching logs indicating a successful connection once "Save & Test" is manually triggered.

logger=tsdb.elasticsearch endpoint=callResource pluginId=elasticsearch dsName=ElasticBFPE dsUID=b80703e4-c812-46a2-8b29-f13654b90b9a uname="juan francisco capote medina" t=2024-11-08T14:17:56.897675656Z level=info msg="Response received from Elasticsearch" statusCode=200 status=ok duration=98.318128ms stage=databaseRequest contentLength=577 resourcePath=
logger=tsdb.elasticsearch endpoint=callResource pluginId=elasticsearch dsName=ElasticBFPE dsUID=b80703e4-c812-46a2-8b29-f13654b90b9a uname="juan francisco capote medina" t=2024-11-08T14:17:57.163600041Z level=info msg="Response received from Elasticsearch" statusCode=200 status=ok duration=57.692506ms stage=databaseRequest contentLength=1240839 resourcePath=banco-pe-omni*/_mapping

Environment:
bitnami/grafana-operator:5.14.0-debian-12-r0
grafana/grafana:11.2.2