Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafka integration #983

Closed
wants to merge 28 commits into from
Closed

Kafka integration #983

Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
7a658de
Add kafka dependency and setup basic demo
TomBebb Oct 21, 2024
dcd496b
Implement basic env var reading / message reading
TomBebb Nov 2, 2024
fb94dca
Support client_id override
TomBebb Nov 2, 2024
8f1bf00
Support int envs to builder mapping
TomBebb Nov 2, 2024
11b8d7d
Support durations
TomBebb Nov 2, 2024
7162580
Handle env parsing error better
TomBebb Nov 2, 2024
ce8d727
Show formatted errors on environment variable parsing error
TomBebb Nov 4, 2024
1d0905e
Sync kafka messages
TomBebb Nov 4, 2024
d708659
Tidy imports
TomBebb Nov 4, 2024
2317da9
Fix double-import
TomBebb Nov 6, 2024
3d2f3b1
Fix metadata error typo
TomBebb Nov 6, 2024
f68b7b1
Add RustRover IDE folder to gitignore
TomBebb Nov 6, 2024
4648715
Merge branch 'main' into kafka-integration
TomBebb Nov 6, 2024
28e316a
Merge branch 'main' into kafka-integration
TomBebb Nov 6, 2024
4299c49
Simplify log push
TomBebb Nov 6, 2024
8c4dbae
Fix kafka order
TomBebb Nov 8, 2024
ed685c6
Improve kafka order
TomBebb Nov 8, 2024
27c3f0b
Support kafka group/partition envs
TomBebb Nov 8, 2024
90a74cc
Simplify parseinterror prefix
TomBebb Nov 8, 2024
229bde8
Support fetch_crc_validation
TomBebb Nov 8, 2024
fc86edc
Support kafka offset storage
TomBebb Nov 8, 2024
1eb536f
Support kafka security
TomBebb Nov 8, 2024
17bdb25
Support kafka fallback offset
TomBebb Nov 8, 2024
0ffc027
Fix security default value
TomBebb Nov 8, 2024
e2c04eb
Merge branch 'main' into kafka-integration
TomBebb Nov 10, 2024
65e38ad
Merge branch 'main' into kafka-integration
TomBebb Nov 11, 2024
0d518a2
Use rdkafka instead of kafka lib
TomBebb Nov 18, 2024
b94a933
Use async message ingestion
TomBebb Nov 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,5 @@ parseable
parseable_*
parseable-env-secret
cache
.idea
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please remove unrelated change


8 changes: 8 additions & 0 deletions .idea/.gitignore
View file
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

102 changes: 102 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions server/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,8 @@ path-clean = "1.0.1"
prost = "0.13.3"
prometheus-parse = "0.2.5"
sha2 = "0.10.8"
kafka = "0.10.0"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using https://github.com/fede1024/rust-rdkafka instead of kafka-rust could allow the implementation to be async

openssl = "0.10.68"

[build-dependencies]
cargo_toml = "0.20.1"
Expand Down
Loading
Loading