Possible data loss?

[selangley]

logstash-input-redis/lib/logstash/inputs/redis.rb

Line 145 in 19927e7

return result

Because the LTRIM is removing the just fetched data from Redis before the results have been processed by Logstash and have had a chance to be put into a Logstash persistent-queue, couldn't this data be lost if Logstash were to unexpectedly go down?

[jordansissel]

Yes. Doing lossless transport through redis is only possible if you do a bunch of extra work to build a lossless abstraction on top of redis. There is no built-in system for this today. Logstash could build this using rpoplpush and a bunch of code to manage state.

…

-Jordan

On Thu, Oct 12, 2017 at 3:41 PM Scott Langley ***@***.***> wrote: https://github.com/logstash-plugins/logstash-input-redis/blob/19927e7b68a5f058f11a3dbace876ba07c9901b9/lib/logstash/inputs/redis.rb#L145 Because the LTRIM is removing the just fetched data from Redis before the results have been processed by Logstash and have had a chance to be put into a Logstash persistent-queue, couldn't this data be lost if Logstash were to unexpectedly go down? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#62>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIC6lSlYoKD1aKfH2u_Evh10jB_InRkks5srpWegaJpZM4P3uv8> .