Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add abiltity to configure error output #668

Merged
merged 91 commits into from
Oct 29, 2024
Merged
Changes from 1 commit
Commits
Show all changes
91 commits
Select commit Hold shift + click to select a range
0b62cff
add abiltity to configure error output
ekneg54 Sep 16, 2024
43dd315
remove store_failed and error_topic
ekneg54 Sep 16, 2024
d4504be
add error queue in pipeline_manager
ekneg54 Sep 16, 2024
28019da
refactor pipeline.py
ekneg54 Sep 18, 2024
1f5317c
add tests for ComponentQueueListener
ekneg54 Sep 19, 2024
0f177fa
add wait for erorr output health
ekneg54 Sep 20, 2024
102d246
remove double log message
ekneg54 Sep 21, 2024
638f256
add test for infinite restart
ekneg54 Sep 23, 2024
915bf6e
adjust examples
ekneg54 Sep 23, 2024
ae0574e
add more result documentation
ekneg54 Sep 23, 2024
fba1f16
fix some tests
ekneg54 Sep 24, 2024
607eef3
move serialisation to enqueue_error
ekneg54 Sep 24, 2024
bc10c7d
add tests for enqueue_error
ekneg54 Sep 24, 2024
a61185d
add developer documentation
ekneg54 Sep 28, 2024
210e013
Refactor Kafka container command in docker-compose.yml
ekneg54 Sep 28, 2024
270b162
Rewrite acceptance tests
ekneg54 Sep 28, 2024
d2d9af3
make hmac not return non_critical_error_message
ekneg54 Sep 28, 2024
de0047f
add error_output_queue
ekneg54 Sep 28, 2024
753b3a4
change order of except blocks in pipeline
ekneg54 Sep 30, 2024
d670bee
change CriticalOutputError message and signature
ekneg54 Sep 30, 2024
48a5ec9
add handling of critical output errors during bulk indexing
ekneg54 Sep 30, 2024
e61f2a1
add event to critical output error raised during opensearch bulk oper…
ekneg54 Sep 30, 2024
4088ad7
add matching errors in enqueue_error
ekneg54 Sep 30, 2024
0ba27e5
add matching errors in enqueue_error
ekneg54 Sep 30, 2024
c5a0c2e
refactor opensearch output connector
ekneg54 Sep 30, 2024
993db77
complete item handling in enqueue_error
ekneg54 Oct 1, 2024
d4b2665
add tests for clearing backlogs
ekneg54 Oct 1, 2024
02e6e4c
fix componentqueuelistener tests
ekneg54 Oct 1, 2024
a9b8e99
ensure raw_input is copied
ekneg54 Oct 1, 2024
915a34e
remove version from compose file due to deprecation warning
ekneg54 Oct 1, 2024
098c9c0
add debugging and additional tests
ekneg54 Oct 1, 2024
6f541cf
fix processor tests in cause of changing processingcrititcalerror
ekneg54 Oct 1, 2024
5061aaa
Working error output start bahavior
ekneg54 Oct 1, 2024
28405a2
fix pipeline_manager tests
ekneg54 Oct 1, 2024
2d5bf03
remove connection between input and output
ekneg54 Oct 1, 2024
ac5c9ca
add tests for shutdown behavior
ekneg54 Oct 1, 2024
c8c7325
update changelog
ekneg54 Oct 1, 2024
85456fc
fix deadlock in pipeline_manager tests
ekneg54 Oct 1, 2024
ccaf128
fix test for kafka output connector
ekneg54 Oct 1, 2024
cbbe7df
fix test for injecting sentinal
ekneg54 Oct 1, 2024
aec7bc4
fix run_logprep tests
ekneg54 Oct 1, 2024
0fc5751
fix linting for Value
ekneg54 Oct 1, 2024
33814c1
remove output from revoke callback
ekneg54 Oct 1, 2024
f17571e
add queuelistener shutdown behavior
ekneg54 Oct 2, 2024
f40bbc9
add tests for shutdown behavior to pipeline_manager
ekneg54 Oct 2, 2024
baadb39
remove opensearch retry tests
ekneg54 Oct 2, 2024
1e1e57f
remove calling batch_finished_callback in store method
ekneg54 Oct 2, 2024
7634333
add opensearch options
ekneg54 Oct 2, 2024
c9feb3f
readd max_retries option
ekneg54 Oct 3, 2024
03e2069
add option to configure error backlog size
ekneg54 Oct 3, 2024
31aa074
add unknown error handling to opensearch output
ekneg54 Oct 3, 2024
0665cfb
fix Changelog
ekneg54 Oct 3, 2024
6e8efec
bump falcon dependency
ekneg54 Oct 3, 2024
26fab9e
make consumer instance id constant for a pipeline process
ekneg54 Oct 9, 2024
4661308
add pid to consumerinstanceid
ekneg54 Oct 9, 2024
81ababa
fix tests for http input and confluent kafka outptu
ekneg54 Oct 9, 2024
0f183a1
fix failed event metric
ekneg54 Oct 9, 2024
2669fee
rewrite failed events metric counting
ekneg54 Oct 10, 2024
2b0465f
add error_output config options to helm chart
ekneg54 Oct 10, 2024
fc81731
add error queue size metric
ekneg54 Oct 10, 2024
04a2373
better opensearch error messages
ekneg54 Oct 14, 2024
fd9ff13
make implementation for queuelistener configurable
ekneg54 Oct 14, 2024
0612bc0
readd event to error event
ekneg54 Oct 15, 2024
01d4651
remove chuck_size from bulk call
ekneg54 Oct 15, 2024
1f5e84d
add error health to healthchecks
ekneg54 Oct 21, 2024
1e150eb
fix typing
ekneg54 Oct 21, 2024
a4a1b23
Refactor test names and improve error logging
dtrai2 Oct 24, 2024
d42b3f0
fix black
dtrai2 Oct 24, 2024
6acb311
refactor and clean up OutputQueueListener
dtrai2 Oct 24, 2024
3ea7916
rename get_component_instance to get_output_instance
dtrai2 Oct 24, 2024
74ff842
remove mathc cases for critical output error handling
dtrai2 Oct 24, 2024
ae1a130
remove unnecessary loop in test
dtrai2 Oct 24, 2024
342156b
remove unused`self.rule = rule` assignment
dtrai2 Oct 24, 2024
d79ca2f
update configuration doc with new parameters
dtrai2 Oct 24, 2024
872bfef
remove error_index key from opensearch_config dict in charts/test_out…
dtrai2 Oct 24, 2024
6e15030
add timeout to error output test
dtrai2 Oct 25, 2024
68d7850
extend error output config test
dtrai2 Oct 25, 2024
af71620
add assertion to verify event _index in message backlog event
dtrai2 Oct 25, 2024
0e945c0
add *args to LogprepException
dtrai2 Oct 25, 2024
8504424
add error handling acceptance test
dtrai2 Oct 25, 2024
59508ed
prevent writing 1 to error output
dtrai2 Oct 25, 2024
8fda657
update error output check in acceptance test
dtrai2 Oct 25, 2024
de06bd6
remove redundant error log statement as it can't be reached
dtrai2 Oct 28, 2024
355fcca
add tests pipeline manager
dtrai2 Oct 28, 2024
d69495f
add test for reading out error from opensearch bulk call
dtrai2 Oct 28, 2024
183fe1a
simplify test for ci pipeline
dtrai2 Oct 28, 2024
29c0611
change loglevel for bypassing rule tree message
dtrai2 Oct 28, 2024
d210506
remove stuck test
dtrai2 Oct 28, 2024
0070857
Refactor ConfluentKafkaInput to store offsets for last message
ekneg54 Oct 28, 2024
99b393b
fix last valid record handling in confluent kafka input tests
dtrai2 Oct 28, 2024
4db7e73
add test to check if error output healthcheck is added to exporter
dtrai2 Oct 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix test for injecting sentinal
ekneg54 committed Oct 9, 2024
commit cbbe7df8cdca2f87bb2dc7fcb7a9f7c26cc4dc6e
2 changes: 1 addition & 1 deletion tests/unit/framework/test_pipeline_manager.py
Original file line number Diff line number Diff line change
@@ -529,7 +529,7 @@ def test_sentinel_breaks_while_loop(self):
listener = ComponentQueueListener(queue, target, output_config)
listener.queue.put(listener.sentinel)
listener._listen()
assert listener.queue.empty()
assert not listener._instance.is_alive()

def test_stop_injects_sentinel(self):
target = "store"