Overview

Looking into ability to show activity in terms of who is accessing dashboard and when. This is the end result of going through the steps below.

Environment

Elastic Cloud version 8.2.0

Steps

Enable Logging and Monitoring

Followed the Enable logging and monitoring documentation to let Elasticsearch Service (ESS) take care of configuring the monitoring agents to ship to a deployment.

Enable Audit Logging

Even though logs are now flowing, you still have to turn on audit logging because it is disabled by default.

This can be done by updating your Kibana settings.

Review Audit Data

One way to look at these logs is through Discover so create a Data View for "elastic-cloud-logs-*" as mentioned in the Enable logging and monitoring documentation. Key fields to look at to filter the noise:

• kibana.saved_object.type: dashboard
• event.action: saved_object_resolve (this was important because before it was listing dashboards I hadn't actually opened)

Great, we can see who is accessing which dashboard and when. To complete the picture, would be nice to know which dashboard and which user instead of just their IDs.

Enriching with Dashboard title

Create a Dashboard Lookup Index

Found the dashboard titles tucked away in the hidden ".kibana" index. As I didn't want to mess around with the index itself, I reindexed it and stashed the dashboard ID into the source of the destination index.

POST _reindex
{
  "source": {
    "index": ".kibana",
    "query": {
      "match": {
        "type": "dashboard"
      }
    }
  },
  "dest": {
    "index": "dashboard-lookup"
  },
  "script": {
    "source": "ctx._source.dashboard_id = ctx._id"
  }
}

Setting Up an Ingest Pipeline

Getting the IDs to Match Up

There was a difference in the identifiers between the ".kibana" index and the audit logs ("cloud-elastic-log-*).

• audit log shows --> "kibana.saved_object.id" : "77949db0-e1a3-11ec-80b0-21a99589f84a"
• kibana index shows --> "_id" : dashboard: 77949db0-e1a3-11ec-80b0-21a99589f84a

To deal with that, added a Script processor to prepend "dashboard" to the ID. Whenever new logs stream in, the ID needed for lookup goes into the dashboard_id field.

String prefix = 'dashboard:';
ctx['dashboard_id'] = prefix + ctx.kibana.saved_object.id

Add an Enrichment Policy

Building on that, this enrichment poicy does a lookup on the dashboard_id and then enriches the data with the dashboard title (e.g. NSF Proposals) that we're looking for.

PUT /_enrich/policy/dashboard-enrich
{
  "match": {
    "indices": "dashboard-lookup",
    "match_field": "dashboard_id",
    "enrich_fields": ["dashboard.title"]
  }
}

Adding that Enrich processor the the ingest pipeline looks like this:

Execute the Enrichment Policy

Don't forget, you have to execute the enrichment policy for it to take effect.

PUT /_enrich/policy/dashboard-enrich/_execute

Updating the Index Template

We want this ingest pipeline to get invoked for any new documents indexed. Add it to the settings on the index template.

Getting Data to Show

You may need to rollover the datastream to start getting the new fields and ingest pipeline humming.

POST elastic-cloud-logs-8/_rollover

Also you could do an update_by_query to update existing data.

POST /elastic-cloud-logs-8/_update_by_query?pipeline=dashboard-enrich
{
  "query": {
    "match": {
      "kibana.saved_object.type": "dashboard"
    }
  }
}

Still need to look into resolving user names but not too shabby...

Limitations

Because the dashboard lookups are reindexed from the .kibana index at a point in time, any new dashboards wouldn't show unless you reindex again.