Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce cursor tracking akin to jdbc input #205

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

jsvd
Copy link
Member

@jsvd jsvd commented May 29, 2024

With this change it's possible to configure a "tracking_field" whose value is injected into the query using the :last_value keyword.
Example configuration:

input {
  elasticsearch {
    id => tail
    hosts => [ 'http://localhost:9200' ]
    index => 'test-alias'
    query => '{ "query": { "range": {"event.ingested": { "gt": ":last_value", "lte": "now"}}}, "sort": [ { "event.ingested": "asc" } ] }'
    tracking_field => "[event][ingested]"
    slices => 2
    schedule => '*/30 * * * * *'
  }
}
output { null {} }

resolves #184
resolves #93

@jsvd jsvd changed the title move cursor tracking to separate class Introduce cursor tracking akin to jdbc input May 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[feature] Support configurations so it behaves like JDBC input Add a "sincedb" type of mecanism
2 participants