Task session_cleanup "session_cleanup" failed: ResponseError: search_phase_execution_exception Root causes: no_shard_available_action_exception

[mikecote]

We are seeing thousands of failures per day on serverless regarding the session cleanup task where it encounters no_shard_available_action_exception errors. While this error is not new, it is negatively impacting the task run success SLO where we have to keep a lower SLO target to accommodate for these failures.

The issue seems to be caused by the reliance of point in time functionality where it needs all shards to be available to succeed. I am opening this issue for the security team to investigate whether we need Elasticsearch to solve these issues or if we can make the task more resilient / not reliant on PIT.

Sample stack trace

ResponseError: search_phase_execution_exception
	Root causes:
		no_shard_available_action_exception: null
    at KibanaTransport._request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:543:27)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at /usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:641:32
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:637:20)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:60:16)
    at ClientTraced.OpenPointInTimeApi [as openPointInTime] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/open_point_in_time.js:56:12)
    at SessionIndex.getSessionValuesInBatches (/usr/share/kibana/node_modules/@kbn/security-plugin/server/session_management/session_index.js:822:9)
    at SessionIndex.cleanUp (/usr/share/kibana/node_modules/@kbn/security-plugin/server/session_management/session_index.js:432:24)
    at TaskManagerRunner.run (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_running/task_runner.js:325:22)

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)