Avro Compact Format

[alexec]

No description provided.

[alexec]

@duglin I've had a negative contributor experience with this PR:

• Fighting the DCO check. There is no need to sign every commit, just one really should enough to meet this check. This is make-work.
• Merge conflicts. I've been editing files that are also edited on master branch. This is also make-work.

This has resulted in a messy PR very unlike my usual work.

[duglin]

@duglin I've had a negative contributor experience with this PR:
* Fighting the DCO check. There is no need to sign every commit, just one really should enough to meet this check. This is make-work.
* Merge conflicts. I've been editing files that are also edited on master branch. This is also make-work.
This has resulted in a messy PR very unlike my usual work.

Signing is a commit-level thing, so the DCO checker needs to check each commit. One option is to --amend your commit so you only have one. No one will complain (or should since I do it and no one has complained).

Not sure what to say about conflicts - that's just part of a collaborative tool like git and I don't see how someone can avoid it if there are busy files. One thing I found helpful is to rebase often. On the positive side, since we only (usually) merge PRs once a week, rebasing shouldn't be needed more than that.

[alexec]

I've traced the cause of the conflict. The instruction for fixing the DCO check are wrong. You cannot push using --force-with-lease once you git merge master, this corrupts the git history.

[alexec]

cloudevents/sdk-java#578

[alexec]

#1210

[alexec]

@duglin open questions:

Existing event formats loose type information (time is rounded to microseconds and offset lost, URIs are de. Both avro and json formats lose URI type information. Ee to limitations in the spec and limitation in JSON respectively.

It is not clear if the HTTP formats support this, but I'm not sure how they would. Say you head custom context attribute that was foo, how can you turn this into a HTTP header and back and preserve type?

Given how common the json format is, it is now de-facto URIs are not supported in custom context attributes.

Avro Turbo chooses performance over complexity. Supporting these types is <= O(9) cost (the number of CE SDKs is 9). Complex systems tend to perform poorly.

For time , the argument is not so strong. we could use an union, but it makes implementation costly, complex and buggy.

My Java implementation is 126 lines long.

• json is 923 lines.
• protobuf is 700 lines.
• xml is 1017 lines long

It is a full order of magnitude smaller. This is achieved by keenly eschewing complexity.

Less code means less bugs. I don't want to make implementing it harder and more buggy than needed.

---

@pierDipi is v3x of the Java SDK backwards incompatible with v2.x? If that is the case, then I'll put this work on hold because we will not have resource to do a version upgrade anytime in the next 3-6 months.

[pierDipi]

@duglin open questions:

Existing event formats loose type information (time is rounded to microseconds and offset lost, URIs are de. Both avro and json formats lose URI type information. Ee to limitations in the spec and limitation in JSON respectively.

It is not clear if the HTTP formats support this, but I'm not sure how they would. Say you head custom context attribute that was foo, how can you turn this into a HTTP header and back and preserve type?

Given how common the json format is, it is now de-facto URIs are not supported in custom context attributes.

Avro Turbo chooses performance over complexity. Supporting these types is <= O(9) cost (the number of CE SDKs is 9). Complex systems tend to perform poorly.

For time , the argument is not so strong. we could use an union, but it makes implementation costly, complex and buggy.

My Java implementation is 126 lines long.

• json is 923 lines.
• protobuf is 700 lines.
• xml is 1017 lines long

It is a full order of magnitude smaller. This is achieved by keenly eschewing complexity.

Less code means less bugs. I don't want to make implementing it harder and more buggy than needed.

@pierDipi is v3x of the Java SDK backwards incompatible with v2.x? If that is the case, then I'll put this work on hold because we will not have resource to do a version upgrade anytime in the next 3-6 months.

In v3, there will be a few breaking changes (current list of breaking changes), however, avro-turbo will be a new module, so we can backport it for a 2.x release

[duglin]

Existing event formats loose type information (time is rounded to microseconds and offset lost, URIs are de. Both avro and json formats lose URI type information. Ee to limitations in the spec and limitation in JSON respectively.

I think I understand how avro loses precision on "time", but how does JSON lose URI type info? Do you mean because it's just a string and there's no way to know if hello is a URI or just a random string? But, also is there a question here?

It is not clear if the HTTP formats support this, but I'm not sure how they would. Say you head custom context attribute that was foo, how can you turn this into a HTTP header and back and preserve type?

Everything is converted to a string. You need to know what the extension's spec-defined type is to know how to deserialize it back to the right type.

Given how common the json format is, it is now de-facto URIs are not supported in custom context attributes.

I'm not sure what you mean by this one.

Avro Turbo chooses performance over complexity. Supporting these types is <= O(9) cost (the number of CE SDKs is 9). Complex systems tend to perform poorly.

For time , the argument is not so strong. we could use an union, but it makes implementation costly, complex and buggy.

My Java implementation is 126 lines long.

* `json` is 923 lines.
* `protobuf` is 700 lines.
* `xml` is 1017 lines long

It is a full order of magnitude smaller. This is achieved by keenly eschewing complexity.

Less code means less bugs. I don't want to make implementing it harder and more buggy than needed.

Is there a question here? It seems to me that if people choose to use Avro then they're accepting the limitations you've mentioned to gain performance.

[alexec]

Thank you. You answered my questions (even if not intentionally).

• URIs custom context attributes not being back to URIs is acceptable.
• Time being truncated to micros is acceptable.
• The main breaking change in the Java SDK appears to be the upgrade to Kafka 3 libraries.

[duglin]

@alexec is this one ready for final review?
Also, in the previous version you updated the SDK.md file - did you want to do that again for this one?

[alexec]

ready for review

[duglin]

Discussed on the 6/15 call and the general direction looks good.
Just a few outstanding comments that need to be dealt with, and some folks wanted a little more time to review.
Hopefully we'll vote next week.

[jskeet]

I agree with the suggestion to rename this to "compact" - I'm happy to do that in an extra commit in this PR if you'd like, so that we're not adding work for you.

Other than that:

• One wording nitpick which can be addressed later
• One question about type differentiation for String/URI/URI-Ref
• One suggested renaming

[jskeet]

Perhaps this should be extensionattributes or extensions? (The other values are all attributes too, apart from data.)

[alexec]

I don't feel strongly - @clemensv _

[alexec]

Changed to “extensions”

[alexec]

I've renamed this to "Avro Compact" because 3 of us preferred that.

[duglin]

Also, in the previous version you updated the SDK.md file - did you want to do that again for this one?

[duglin]

can you rebase? There's a conflict on the SDK.md

[alexec]

there is always a conflict with that file, I'll fix conflicts once approved

[duglin]

I fixed the rebase for you so we can have the checker do it's job

[duglin]

I also added a change to the primer since the Adobe docs moved. Might as well just reuse this PR. Now the only failing check is an ok one.

[duglin]

@alexec sorry for the late notice, but I just realized that for consistency I think you should put the new spec into: cloudevents/working-drafts dir, like we did for XML.

[JemDay]

Minor tweak ....

[jskeet]

Ping @clemensv for the naming of "attributes" vs "extensions" for the map of extension attributes. I'd personally like to see that resolved before merging. One new thing I've noticed as well around nullability... but we're getting there :)

[jskeet]

Thanks for being patient with us :)

[duglin]

@alexec can you please move the new spec into the cloudevents/working-drafts dir?

[duglin]

@alexec couple of bad refs now:

cloudevents/SDK.md: 'https://github.com/cloudevents/spec/blob/main/cloudevents/formats/avro-compact-format.md' was not found
cloudevents/working-drafts/avro-compact-format.md: Translation file cloudevents/languages/he/working-drafts/avro-compact-format.md does not exist
cloudevents/working-drafts/avro-compact-format.md: Translation file cloudevents/languages/zh-CN/working-drafts/avro-compact-format.md does not exist

actually, I think the first one is supposed to fail but it needs to point to "working-drafts" instead of "formats". The other 2 probably need the files moved to the working-drafts dir

[duglin]

@alexec thanks - merging as it was approved during last week's call.