Fix: filtered records holding up pipeline with destination batching

[hariso]

Description

In #1986 we discovered an issue where, if there are records to be filtered and batching with a time delay is used, Conduit is waiting for the batch delay as soon as it encounters a filtered record.

The underlying issue is the following: when a processor node in a pipeline identifies a filtered record, then it goes on to acknowledge it. However, the acker node will not acknowledge it until the previous records have been acknowledged.

If batching is enabled, and the batch is not full, then Conduit is waiting for the batch to fill up, to flush it and ack the records. This creates a cyclic dependency: the filtered record is waiting for the previous records to be acked (the batch flushed), and the previous records are waiting for the filtered record to be acked, so that new records can come in and form a full batch.

This issue is solved by not nacking the filtered records in the processor. Instead, we let them pass through the pipeline until they reach the final node, the destination record, which simply acks them.

Fixes #1986.

Quick checks

• I have followed the Code Guidelines.
• There is no other pull request for the same update/change.
• I have written unit tests.
• I have made sure that the PR is of reasonable size and can be easily reviewed.

[raulb]

Code looks good.

[hariso]

@raulb @maha-hajja Thanks for the review and the suggestions! I've updated the PR with a few more tests.

I've also tested it with a few pipeline:

• file-to-file, without processor, without filter, with batching
• file-to-file, with processor, no filter, with batching
• file-to-file, with processor and filter, with batching
• file-to-file, with processor and filter, with batching

I've tested all of them with and without batching (batch size + batch delay).

[maha-hajja]

nice work!