Skip to content

Error handling deleted the original content & added new Processor to convert protobuf to Avro #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 23 commits into
base: master
Choose a base branch
from
Open

Error handling deleted the original content & added new Processor to convert protobuf to Avro #10

wants to merge 23 commits into from

Conversation

benyaa
Copy link

@benyaa benyaa commented Dec 15, 2021 •
edited
Loading

Hi,
It seems that the current error handling resets the FlowFile's original content..
So:

  1. I fixed it by throwing exception so the original FlowFile won't be rewritten and then transfer it to failure.
  2. I added a capability to preserve the field names
  3. I added a capability to process messages separated by a demarcator(like when consuming using the ConsumeKafka processor)
  4. I added a new processor that converts the Protobuf messages into Avro messages so they would be easier to handle with NiFi(rather than JSON that doesn't preserve the original schema)
  5. I changed the processors' names to EncodeProtobuf and DecodeProtobuf to match NiFi conventions

Hope it's good enough!

benyaa and others added 23 commits August 8, 2019 14:19
@benyaa
upgraded protobuf & nifi libraries
22b4522
@benyaa
1. Changed name to AbstractProtobufProcessor so it would fit the conv…
65640a5 
…ention

2. moved setUpSchema to OnScheduled so it could evaluate expression language from variable registry
3. Added documentation and added new property PROTOBUF_MESSAGE_TYPE to allow default message type.
@benyaa
Updated tests to use properties by their variables.
4331d3a
@benyaa
Added docs and added expression evaluation to compile schema
6b8ab36
@benSolar
Renamed processors to match NiFi conventinos
749ed5d
@benyaa
added validation for data arrival
8ee2570
@benyaa
Merge branch 'master' into develop
fc78e25
@whiver
Merge pull request whiver#4 from benyaa/develop
a27324e 
- upgraded protobuf & nifi libraries
- Changed name to AbstractProtobufProcessor so it would fit the conv…
- Updated tests to use properties by their variables.
- Added docs and added expression evaluation to compile schema
@benyaa
removed unnecessary schema recompile(since we do it as soon as it is …
c27c6f1 
…configured) and added option to batch process messages with a demarcator
@benyaa
removed unnecessary parameters
87df7ae
@benyaa
Fixed error handling to not drop flowfile content on error
0e77cff
@benyaa
added new processor to convert protobuf to avro format
8bce8c4
@benyaa
added new processor to list
f80158f
@benyaa
removed unused imports
021d37d
@benyaa
switched to using a StreamDemarcator so not all messages would be loa…
fd71a54 
…ded to memory at once
@benyaa
added option to encode json array to demarcated protobuf messages
9dbeb05
@benyaa
fixed error handling in ConvertProtobufToAvro
a5f7d35
@benyaa
fixed thread synchronization
744110d
@benyaa
added basic support for avro to protobuf
08db46e
@benyaa
changed ConvertAvroToProtobuf to ConvertRecordToProtobuf to support a…
2fa74b9 
…ny supported NiFi format
@benyaa
added javadoc
2420c84
@benyaa
added support for single-field MESSAGE fields
5d635d7
@benyaa
Merge branch 'whiver:develop' into develop
1d02858
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@benyaa @benSolar @whiver