Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhancement(new sink): new sink for Google chronicle #13550

Merged
merged 39 commits into from
Jul 22, 2022
Merged
Changes from 1 commit
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
c4ca4e3
Add initial sink
StephenWakely May 12, 2022
d209314
Started on batching and partitioning
StephenWakely May 13, 2022
9fb3576
It works old style.
StephenWakely May 19, 2022
c7ad1cd
New style sink working.
StephenWakely May 19, 2022
36aa6cb
Encode data for unstructured.
StephenWakely May 25, 2022
62a192e
WIP
StephenWakely Jun 20, 2022
07a798f
Fixed endpoint.
StephenWakely Jun 22, 2022
ca2bf1f
Request should be post
StephenWakely Jun 26, 2022
5cc9e98
Tidy code
StephenWakely Jul 1, 2022
61f62f8
Added content-type header
StephenWakely Jul 5, 2022
2c36d89
Added integration test.
StephenWakely Jul 13, 2022
6e3469e
Merge remote-tracking branch 'origin' into stephen/chronicle_sink
StephenWakely Jul 13, 2022
c0c7f4f
Fixed merge
StephenWakely Jul 13, 2022
f635f86
Fixed encoding.
StephenWakely Jul 14, 2022
f4fc89d
Use generated auth keys.
StephenWakely Jul 14, 2022
99ad86e
Remove old chronicle folder
StephenWakely Jul 14, 2022
edcda8d
Remove commented module.
StephenWakely Jul 14, 2022
0372922
Tidy module comments.
StephenWakely Jul 14, 2022
9d2b2c4
Merge remote-tracking branch 'origin' into stephen/chronicle_sink
StephenWakely Jul 14, 2022
2f8aa6e
Ackable is no more.
StephenWakely Jul 14, 2022
fdbb982
Fixed unit tests.
StephenWakely Jul 14, 2022
89ee257
Clippy
StephenWakely Jul 14, 2022
12e1dd6
Merge remote-tracking branch 'origin' into stephen/chronicle_sink
StephenWakely Jul 14, 2022
de4841f
Feedback from Pablo.
StephenWakely Jul 14, 2022
b8ecf2a
Added newline
StephenWakely Jul 15, 2022
fd8bb8f
Fmt
StephenWakely Jul 15, 2022
de20f2b
Merge remote-tracking branch 'origin' into stephen/chronicle_sink
StephenWakely Jul 15, 2022
3e621f4
Add docs for google chronicle unstructured sink.
StephenWakely Jul 18, 2022
9dcaac6
Remove Framing from Encoder config.
StephenWakely Jul 18, 2022
d875978
Address feedback.
StephenWakely Jul 18, 2022
f2067d1
Cue fmt
StephenWakely Jul 18, 2022
d09ae26
Feedback from Kyle.
StephenWakely Jul 20, 2022
f2607d8
Compression is not an option.
StephenWakely Jul 21, 2022
15c95af
Update goauth.
StephenWakely Jul 21, 2022
25416e0
Add integration tests for the unhappy paths.
StephenWakely Jul 21, 2022
ce19b14
Clippy
StephenWakely Jul 21, 2022
f375630
Add compression:enabled to docs.
StephenWakely Jul 22, 2022
f61889f
cue fmt
StephenWakely Jul 22, 2022
5e58b30
Write using as_tracked_write to save an allocation.
StephenWakely Jul 22, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Use generated auth keys.
Signed-off-by: Stephen Wakely <[email protected]>
  • Loading branch information
StephenWakely committed Jul 14, 2022
commit f4fc89d6268f7e5588305ceca1c2ab2b7e479fe5
12 changes: 12 additions & 0 deletions scripts/integration/chronicleauth.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"type": "service_account",
"project_id": "chronicle-test",
"private_key_id": "1",
"private_key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKQIBAAKCAgEAw/xPtXSFlJgx7qT8C/xNAT/zJRCm854rdw0+0X/m2UnWVLBC\nd92AE8hSIvQCI0wz7y300L0pyUzsHcAw3PA+LDKhjiNKWDlWQsSZ10HpHcmJ9ByM\nxv/gKh5ocgeIzOmycGId9wvGdHd4eZxL86FqN7Ezkixh05lP1Xg5fNNalF+2iWj5\nq9SlI5sBIzbtngtph4Jp9h6g7JaWVcoR2VcAbNK1Hdp1aanbqgroSsRWFJ91NtEK\nq+XzaNpPIXcM9wz1dKAOB34yMlig1UuOZyEqBPZJvXjh/N9TdG6ZTjA7LxvKyGOT\nYHBfIfZgint/UTCN72Qg3xN7f7E4EEWD7jMXdhPm1ERCEOjRlobMj1xw/spFH4O+\nxLAHHp1g4pWM1wTU2BHJH9yqr3GhMghnxQs4H56fx/ML1+onPJqwDzgMiFi5Aqmt\n5Jze5zBfLflUdrcvbeJ5ea5RQLK/f/MuKI2AuFXwXVDqQ86lQ0IJMuTrZBFMjOSq\nInTGkp71zetZ7pXs1r5pP5+aAPtClLn72HRfRuVFALkdXi/lOAQbwuKoLmPSjLAc\nyiPY9fUPlmtDOywTdO81b8tQG8ejSxIqDHd52bhTt5hi5TrzqCa9QIGQOtUJdCSS\nCKIUdTWdcDDyE5PGut/9E+Nd9i3syxfMapOkkM7swbKRjeSKOLNwur6oQusCAwEA\nAQKCAgBUCCcVInT0FZ2zOUJo0TEjTAww5EbxRexVC3TX1wNgP2yKu1iElJCJ6uZs\n3jrcb5wE/atupOz5xuWmi/VnhgriYIMssqz5zvvRfQKJbVVK1FM2O9eQq72t0YPv\nAJQDaGB+F1PJzV62KVOasQ1P1PH6FxyJnEYFuEKmwPnFL8oyvCdEG6KRrodwENIY\nC0Eu9q200JImWVOenaKv/ghRLCCeiGnn9vdI6u+opSycgjp2dTiI3HC668nGN3Se\nYLFKbGACNW5OgCl9tMqhKdhRF8MwhXlwqBgdmJPr5FAPWlbGCCCWczuJqiTgJweV\n2QXqBeuVrVT60Ruu0Im9UFEVHQr9ccvXg2tAy6p3WRq+37phr3RaeVmiZDc5GkC0\nn/1UeSrcIICO1kZKHpuPSjcgty90kXKQdCVNzbHxILND9FFzOBilDSWhdaeGZ+x+\nC+C8pVJeMrVnUqNDsd6hH4uNJQuTR/wUAPYGcW3CPVpy7l+UyoB3hZPVBgZkt9Fa\nCjK6+Wo4aohcdl9/Mk/R8XTeSXT9L38DmFgfCxnV0v9vYUCK6mcxurAxtQ6sNO2i\nuz79gNEhXEAcgarKrF0V/EOfl+uQfB8DEuuatkrw65FEmK//fY1ULI0jv7KFhKC/\njUgIi5p7CTfDXlRsUcFlr/qNHG0O8HvC+nW6zTonA//epfenoQKCAQEA4gq+7g+0\nIjPdrKaXL3n23sgsxzZp5GtedfPH51MaaH93cFhiTSQSiXXGYIbg9KzfNtqX4IiJ\njBk2IVQx2h5T40/d+GGQN03+70gQ5NWycPH9vRFR6mHQR1T1luqEg/7mYQydvTg7\n1KnhUS/lPsEapheztljVKv/cMxgL9qDAk1HoqBAxlgbu6ya6hp06n9vFTUKfg598\nxvZ11R/TUxlCBcnieGPjzsO+qXopepJUarlleN+LhvlOySGvzlTu78/10aKDrMHR\nRG3GTBKLuDQfSI6+diSIpA/FqW5b2uFc4nnBcf+gQt9PPtEReFNjqOXD/Os8Ey1a\nVRI/kyqtBM2gRwKCAQEA3fXMiKSH7qKmkAH3NKSD1wakaJt9Gy5zgSoip6X2kOWi\n/vqalnB9TdynVJaPfJ1K0b/Zm+Fqi658ql5o3jDljJavOdsL4pZMzP+XTP2BNCM9\nvO7u4VHh2dk4yPZhBCTIVxF7dW5BfF31kpGoPA/NNTm2iwe0CSJ7sb3kGwQg9n7t\nK14OcyUvkFrIu9ELY0wXZkPfe9zHeIWB92ZSmhcs/MC6hrWdzYSRrFreFRSGaAKe\nMK7NLvRVVCU+Ln/qvCWReUzQfbKqDB4ayf5DulcvuRzI57znHQur3/EXV6LTmATx\nO6zykbzn9jwzCGKRuytv9DMfmm3Pdzn7/D5NzzlePQKCAQEAk5X8j6cFMAe5Rs8G\ni2MQwujkA7YNcayA4nanIOxro6sGKv4qQL+6srGJMuEd4+MzYrdazqmXVTfo1d4J\nJqcCkrpE7bWV2eY8+7KMBT1Z4tC9oUK7v1LaEJjyOPUqTJyTgu4f17ntpq68hKvG\n/DBM3AFBv19E32xrM5kRxpey5P1n8MOR/KjOOg7xXo8uBc4Shp1nVSM+7xFgYiUD\nb5BgQTNNgNmHlirMRA+VIYiFQaGHFWKv0RLaVAyAozFlSIL5t0RqwNNTdWBPRsAd\nJcoyWseIp862wJfjsPpzUvguE1j36xzCYKc5MISE6ZFY0qAiVPKqKMNF5aGijM4i\nZa50ZQKCAQEAhmXmUjrmAC5Gkt5N1HdlSVdBSAIWj9XhpK1hzIems/gIaUJPRpaY\nmQPp4kNEYN1RR45WcpfcjvW5kOOXtip/ZFB9i5fCK5J8sIh3EexSRsKsCHHY/WA6\nVFG+m5jsB7lKAcaZLfLz/h1arEQ8TRn4VKk8ihTBA+L4aRilPWKTnw4eB+ts7nXu\nxgbiky8yKQ8TAB5K/VIjCiw1FRvFGn68A9OjXhwskBwcpPNkk/kYpXQoWywNG8hg\nvpHKhBd6iDBrVLqZZ/M+GhZEfFjUo8lVIU53rDJq8uB1UyVF0A/M9bjQZSUEwBBV\nmHi9Ie+ldGMJ6CDomvaw7+2RUhxbuwVutQKCAQAOj1TP9RbwkkYI7sZf0BV+ppx2\nhr77O14sIzxZ3/zFREbH4vw48GJOoj8nc3opYQNnUiZwtat3ptnou+1zprX7koTQ\n25rdoCfjrtbw4MUawOV3LvdWVVv5NCmScEyzM+xP8h6aRWQKDxWUYfaGOVBHeuF5\n2g4oSR4s+h1NJvslREr3RiR/m7jbL6VDQhoOUh6o9/bSJW0AqE6Lsx9ws41h22kh\n20Kl6W4adF9XDR9OsT5dznio2gYr2EPt+PTLocPk7s5s7thdWS3oUQ7UA8XlAG6h\nZzT9O+qtMKELxwOgB25fMBcg3kZFHVyqrCQQJSaP82FYdbIHbWiy6VuBWPbd\n-----END RSA PRIVATE KEY-----\n",
"client_email": "[email protected]",
"client_id": "1",
"auth_uri": "http://chronicle-emulator:3000/o/oauth2/auth",
"token_uri": "http://chronicle-emulator:3000/token",
"auth_provider_x509_cert_url": "https://chronicle-emulator:3000/oauth2/v1/certs",
"client_x509_cert_url": "https://www.ohno"
}
13 changes: 13 additions & 0 deletions scripts/integration/chroniclepub.pem
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
-----BEGIN RSA PUBLIC KEY-----
MIICCgKCAgEAw/xPtXSFlJgx7qT8C/xNAT/zJRCm854rdw0+0X/m2UnWVLBCd92A
E8hSIvQCI0wz7y300L0pyUzsHcAw3PA+LDKhjiNKWDlWQsSZ10HpHcmJ9ByMxv/g
Kh5ocgeIzOmycGId9wvGdHd4eZxL86FqN7Ezkixh05lP1Xg5fNNalF+2iWj5q9Sl
I5sBIzbtngtph4Jp9h6g7JaWVcoR2VcAbNK1Hdp1aanbqgroSsRWFJ91NtEKq+Xz
aNpPIXcM9wz1dKAOB34yMlig1UuOZyEqBPZJvXjh/N9TdG6ZTjA7LxvKyGOTYHBf
IfZgint/UTCN72Qg3xN7f7E4EEWD7jMXdhPm1ERCEOjRlobMj1xw/spFH4O+xLAH
Hp1g4pWM1wTU2BHJH9yqr3GhMghnxQs4H56fx/ML1+onPJqwDzgMiFi5Aqmt5Jze
5zBfLflUdrcvbeJ5ea5RQLK/f/MuKI2AuFXwXVDqQ86lQ0IJMuTrZBFMjOSqInTG
kp71zetZ7pXs1r5pP5+aAPtClLn72HRfRuVFALkdXi/lOAQbwuKoLmPSjLAcyiPY
9fUPlmtDOywTdO81b8tQG8ejSxIqDHd52bhTt5hi5TrzqCa9QIGQOtUJdCSSCKIU
dTWdcDDyE5PGut/9E+Nd9i3syxfMapOkkM7swbKRjeSKOLNwur6oQusCAwEAAQ==
-----END RSA PUBLIC KEY-----
8 changes: 4 additions & 4 deletions scripts/integration/docker-compose.chronicle.yml
Original file line number Diff line number Diff line change
@@ -2,12 +2,12 @@ version: "1"

services:
chronicle-emulator:
image: chronicle-emulator:latest
image: docker.io/plork/chronicle-emulator:latest
ports:
- 3000:3000
volumes:
- /home/plertrood/src/rust/chronicle-emulator/public.txt:/public.txt
command: [ "-p", "/public.txt" ]
- ${PWD}/scripts/integration/chroniclepub.pem:/public.pem
command: [ "-p", "/public.pem" ]
runner:
build:
context: ${PWD}
@@ -35,7 +35,7 @@ services:
- ${PWD}:/code
- cargogit:/usr/local/cargo/git
- cargoregistry:/usr/local/cargo/registry
- /home/plertrood/src/timber/confs/testchronicleauth.txt:/testchronicleauth.txt
- ${PWD}/scripts/integration/chronicleauth.json:/chronicleauth.json

volumes:
cargogit: {}
2 changes: 1 addition & 1 deletion src/sinks/gcp/chronicle_unstructured.rs
Original file line number Diff line number Diff line change
@@ -455,7 +455,7 @@ mod integration_tests {
indoc! { r#"
endpoint = "{}"
customer_id = "customer id"
credentials_path = "/testchronicleauth.txt"
credentials_path = "/chronicleauth.json"
log_type = "{}"
encoding.codec = "text"
"# },