[8.x](backport #41446) [filebeat] Elasticsearch state storage for httpjson and cel inputs

[mergify]

Proposed commit message

[filebeat] Elasticsearch state storage for httpjson input

This is a POC for Elasticsearch as State Store Backend for Security Integrations for Agentless solution.

The scope of this change was narrowed down to supporting only httpjson inputs in order to support Okta integration for the initial release. All the other integrations inputs still use the file storage as before.
This is a short term solution for the state storage for k8s environment.

This is the first cut and the details can change depending on the feedback.

Current feature currently could be enabled AGENTLESS_ELASTICSEARCH_STATE_STORE_ENABLED, to be decided how this would be configurable in k8s.

This change currently contains the hacky approach to the AGENTLESS_ELASTICSEARCH_APIKEY overwrite. This allows to the user to provide the ApiKey with elevated permissions that are required in order to be able to create/write/read the state index per input. THIS IS FOR DEVELOPMENT/TESTING ONLY. REMOVE BEFORE THE MERGE.

The existing code relied on the inputs state storage to be fully configurable before the main beat managers runs. The change delays the configuration of httpjson input to the time when the actual configuration is received from the Agent.

There is an assumption that the index template for the state storage indices is already in place before the storage is used

PUT _index_template/agentless_state_template
{
  "index_patterns": [
    "agentless-state-*"
  ],
  "priority": 300,
  "template": {
    "mappings": {
      "properties": {
        "v": {
          "type": "object",
          "enabled": false
        },
        "updated_at": {
          "type": "date",
          "format": "strict_date_optional_time||epoch_millis"
        }
      }
    },
    "settings": {
      "number_of_shards": 1
    }
  }
}

Example of the state storage index content for Okta integration:

{
  "took": 6,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "agentless-state-httpjson-okta.system-028ecf4b-babe-44c6-939e-9e3096af6959",
        "_id": "httpjson::httpjson-okta.system-028ecf4b-babe-44c6-939e-9e3096af6959::https://dev-36006609.okta.com/api/v1/logs",
        "_seq_no": 39,
        "_primary_term": 1,
        "_score": 1,
        "_source": {
          "v": {
            "ttl": 1800000000000,
            "updated": "2024-10-24T20:21:22.032Z",
            "cursor": {
              "published": "2024-10-24T20:19:53.542Z"
            }
          }
        }
      }
    ]
  }
}

The naming convention for all state store is agentless-state-<input id>, since the expectation for agentless we would have only one agent per policy and the agents are ephemeral.

Currently in order to run the agent with Elasticsearch state storage a couple of environment variables would be required:

sudo AGENTLESS_ELASTICSEARCH_STATE_STORE_ENABLED=1 AGENTLESS_ELASTICSEARCH_APIKEY=xxxxxxxx-xvpDXfB:jVMRsW7SRIxxxxxxxxx ./elastic-agent -e

where the ApiKey in the

DEPENDENCIES / TODOS:

• Approval of teams for this approach
• Kibana (?) side change is required for the agentless-state index template boostrapping
• Kibana or the intergration package (or both) change is required in order to include the permissions for agentless-state- with the Elasticsearch ApiKey (Remove the hack). I suspect that Kibana fleet code could be modified to recognize agentless supporting integration and include the proper index name for the agentless-state for the ApiKey permissions.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

The change should have no impact, and without the feature enabled the filebeat should work as before using the file system storage for the state.

---

This is an automatic backport of pull request #41446 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 8180f23 has failed:

On branch mergify/bp/8.x/pr-41446
Your branch is up to date with 'origin/8.x'.

You are currently cherry-picking commit 8180f23fb.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   filebeat/beater/store.go
	new file:   filebeat/features/features.go
	new file:   filebeat/features/features_test.go
	modified:   filebeat/input/filestream/environment_test.go
	modified:   filebeat/input/filestream/input_test.go
	modified:   filebeat/input/filestream/internal/input-logfile/manager.go
	modified:   filebeat/input/filestream/internal/input-logfile/store.go
	modified:   filebeat/input/filestream/internal/input-logfile/store_test.go
	modified:   filebeat/input/journald/environment_test.go
	modified:   filebeat/input/journald/input_filtering_test.go
	modified:   filebeat/input/v2/input-cursor/manager.go
	modified:   filebeat/input/v2/input-cursor/store.go
	modified:   filebeat/input/v2/input-cursor/store_test.go
	modified:   filebeat/registrar/registrar.go
	modified:   libbeat/statestore/backend/backend.go
	new file:   libbeat/statestore/backend/es/error.go
	new file:   libbeat/statestore/backend/es/notifier.go
	new file:   libbeat/statestore/backend/es/notifier_test.go
	new file:   libbeat/statestore/backend/es/registry.go
	new file:   libbeat/statestore/backend/es/store.go
	modified:   libbeat/statestore/backend/memlog/store.go
	modified:   libbeat/statestore/mock_test.go
	modified:   libbeat/statestore/store.go
	modified:   libbeat/statestore/storetest/storetest.go
	modified:   x-pack/filebeat/input/awss3/states.go
	modified:   x-pack/filebeat/input/awss3/states_test.go
	modified:   x-pack/filebeat/input/salesforce/input_manager_test.go
	modified:   x-pack/filebeat/tests/integration/managerV2_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   filebeat/beater/filebeat.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[cla-checker-service]

💚 CLA has been signed

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[elasticmachine]

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)