Consumer Filtering Performance

[baldugus]

Hi nats-io team,

I thought of using NATS with multiple streams consuming overlapping subjects, but soon realized the right way to do it is by having a big stream with multiple consumers filtering the subjects by what they want.

My concern with that model is that, in my usage i need flexibility over the way topics are viewed, and i might need to replay them oftenly, so i don't know if as data grows, this filtering might get slow as it would need to skip messages not matching it.

For example, if i created a stream consuming the subject foo.bar.> with a million messages on it and now i needed to create a new consumer filtering subject foo.bar.baz.xyz but it has only one message matching it in the middle of the stream, would it go through the whole stream to find this message? Does it have some sort of index which would reduce the impact in similar scenarios?

Thanks in advance for the answer and all the time spent developing this awesome software.

[derekcollison]

We have put alot of work into making that efficient. Try out the 2.9 RC via synadia/nats-server:nightly and test out to verify.

[jnmoyne]

To expand on Derek's answer: no the sever doesn't do like a table scan over all of the messages in the stream and things indexing are used to make operations to find the first and last message(s) in a stream very efficient indeed.

[baldugus]

Thanks @derekcollison and @jnmoyne for the quick and effective replies. I'm yet to mark this as the answer because i didn't find the time to benchmark the 2.9 RC.

After i'm able to do it, i'll complement my question with the tests results and close this. Again, thanks very much for the help.