Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[filebeat] Elasticsearch state storage for httpjson and cel inputs #41446

Merged
merged 77 commits into from
Jan 28, 2025
Merged
Changes from 1 commit
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
55c72d3
[filebeat] Elasticsearch state storage for httpjson input
aleksmaus Oct 24, 2024
1bf288d
Fixup tests
aleksmaus Oct 24, 2024
e003053
Linter
aleksmaus Oct 25, 2024
dfce978
Enabled elastisearch storage support for cel input and some cleanup
aleksmaus Oct 28, 2024
e2e25fa
Remove the "hack" with .Each implementation
aleksmaus Oct 30, 2024
953355b
Merge branch 'main' into poc/es_state_store
aleksmaus Oct 31, 2024
c1fc2a8
Adjust for the latest main es client signature change
aleksmaus Oct 31, 2024
c9b0256
Make check happy
aleksmaus Oct 31, 2024
21d451d
Fixed missing interface method on test mock store
aleksmaus Oct 31, 2024
ffb9364
Add error check in ES store Each
aleksmaus Oct 31, 2024
10d212f
Parameterize the supported input types through environment variables
aleksmaus Nov 5, 2024
24000d7
Delete the dev tests file
aleksmaus Nov 5, 2024
a0d019e
Address code review comments
aleksmaus Nov 7, 2024
34003d4
Add refresh=wait_for for data consistency
aleksmaus Nov 8, 2024
36e9983
Revert "Add refresh=wait_for for data consistency"
aleksmaus Nov 18, 2024
aee4112
Cleanup
aleksmaus Nov 18, 2024
76b11a4
Added updated_at field
aleksmaus Nov 18, 2024
1fe1c0e
Merge branch 'main' into poc/es_state_store
aleksmaus Nov 19, 2024
3c5bab1
Merge branch 'main' into poc/es_state_store
aleksmaus Nov 25, 2024
4ac1bd0
Eliminate AGENTLESS_ELASTICSEARCH_STATE_STORE_ENABLED setting. The El…
aleksmaus Nov 25, 2024
535560e
Merge branch 'main' into poc/es_state_store
aleksmaus Dec 2, 2024
4940097
Merge branch 'main' into poc/es_state_store
aleksmaus Dec 6, 2024
e73a1b7
Added logging per code review request
aleksmaus Dec 9, 2024
6c39fd0
Some cleanup and notifier utz
aleksmaus Dec 10, 2024
716758f
Add some integration testing coverage
aleksmaus Dec 10, 2024
5ec7ed4
Merge branch 'main' into poc/es_state_store
aleksmaus Dec 11, 2024
904d8e8
Merge branch 'main' into poc/es_state_store
aleksmaus Jan 7, 2025
02d684a
Merge branch 'main' into poc/es_state_store
aleksmaus Jan 7, 2025
5a9cdb5
Fix the Filebeat test harness for agentbeat tests
aleksmaus Jan 7, 2025
fd4cd00
Fixed the imports formatting for the notifier_test
aleksmaus Jan 7, 2025
8dd3938
Merge branch 'main' into poc/es_state_store
aleksmaus Jan 8, 2025
c52bd65
Merge branch 'main' into poc/es_state_store
aleksmaus Jan 9, 2025
8779697
Merge branch 'main' into poc/es_state_store
aleksmaus Jan 13, 2025
c195cae
Remove loading API key through envvar workaround
orestisfl Jan 17, 2025
dfafeff
remove os import
orestisfl Jan 17, 2025
e192747
nitpick
orestisfl Jan 17, 2025
71fd5dc
Use simple Mutex
orestisfl Jan 21, 2025
d4bc727
Notifier: add docs
orestisfl Jan 21, 2025
5569e7f
integration test: check elasticsearch is used
orestisfl Jan 22, 2025
a6fbdb3
Merge branch 'main' into poc/es_state_store
orestisfl Jan 22, 2025
aa96d56
revert loading env in NewFilebeat()
orestisfl Jan 22, 2025
71901b0
Remove Env
orestisfl Jan 22, 2025
970b878
typo
orestisfl Jan 22, 2025
28a8332
HandlerFunc: check error
orestisfl Jan 22, 2025
426c681
Fix log-check
orestisfl Jan 22, 2025
b15b0bb
Re-use existing debug log
orestisfl Jan 22, 2025
3e2aed3
re-use function
orestisfl Jan 22, 2025
89c6e9c
checkFilebeatLogs helper
orestisfl Jan 22, 2025
fa8742e
Make test check store initialization
orestisfl Jan 23, 2025
bdf0777
Debug-log inputID
orestisfl Jan 23, 2025
b71b11e
Introduce var inputUnit
orestisfl Jan 23, 2025
3d11005
UUID input id
orestisfl Jan 23, 2025
6b98fab
Check that zero and then one key is read
orestisfl Jan 23, 2025
af14d49
Check inputID is configured
orestisfl Jan 23, 2025
4fc4566
output unit func
orestisfl Jan 23, 2025
3e859b8
final atomic.Bool
orestisfl Jan 23, 2025
3041e60
Merge remote-tracking branch 'origin/main' into poc/es_state_store
orestisfl Jan 23, 2025
083fd86
comment
orestisfl Jan 23, 2025
2e173a4
Change uuid library
orestisfl Jan 23, 2025
c9ebe0f
Merge branch 'main' into poc/es_state_store
orestisfl Jan 24, 2025
c69aefe
check number of http calls
orestisfl Jan 23, 2025
f49cbfd
typo
orestisfl Jan 27, 2025
0cb23d7
integration test: Add more elaborate checks
orestisfl Jan 27, 2025
fe16a10
Fix delta
orestisfl Jan 27, 2025
89ac0bf
Merge branch 'main' into poc/es_state_store
orestisfl Jan 27, 2025
8770e00
Revert input-logfile changes
orestisfl Jan 27, 2025
cef8721
fix store_test.go
orestisfl Jan 27, 2025
a988659
reduce debug-log
orestisfl Jan 27, 2025
51cb319
avoid return error
orestisfl Jan 27, 2025
8d362da
cleanup
orestisfl Jan 27, 2025
59c4d52
notifier: Call listener with last config
orestisfl Jan 28, 2025
5b27a39
Simplify test
orestisfl Jan 28, 2025
040f3c5
Merge branch 'main' into poc/es_state_store
orestisfl Jan 28, 2025
c490984
nitpick
orestisfl Jan 28, 2025
1b7842e
sanity
orestisfl Jan 28, 2025
9e4c86e
Add unit tests for feature flag
orestisfl Jan 28, 2025
c1f5971
godoc
orestisfl Jan 28, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
godoc
orestisfl committed Jan 28, 2025
commit c1f5971bb69b4125608e2ee4337115c5ba06ad54
2 changes: 2 additions & 0 deletions filebeat/beater/filebeat.go
Original file line number Diff line number Diff line change
@@ -82,6 +82,8 @@ type Filebeat struct {
type PluginFactory func(beat.Info, *logp.Logger, StateStore) []v2.Plugin

type StateStore interface {
// Access returns the storage registry depending on the type. This is needed for the Elasticsearch state store which
// is guarded by the feature.IsElasticsearchStateStoreEnabledForInput(typ) check.
Access(typ string) (*statestore.Store, error)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Godoc for this parameter?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest something like

	// Access returns the storage registry depending on the type.
	// The value of typ is expected to have been obtained from
	// [cursor.InputManager.Type] and represents the input type.

CleanupInterval() time.Duration
}