Skip to content

Commit

Permalink
feat(#234): remove old kafka config and update README
Browse files Browse the repository at this point in the history
  • Loading branch information
@StephanHCB
StephanHCB committed Nov 24, 2023
1 parent 0d689c2 commit 5a69d44
Show file tree
Hide file tree
Showing 8 changed files with 45 additions and 94 deletions.
46 changes: 40 additions & 6 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -58,10 +58,7 @@ the [`local-config.yaml`][config] can be used to set the variables.
| `GIT_COMMITTER_NAME` | | Name of the user used to create the Git commits. |
| `GIT_COMMITTER_EMAIL` | | E-Mail of the user used to create the Git commits. |
| | | |
| `KAFKA_USERNAME` | | Leave ALL of the following `KAFKA_` fields empty to skip the Kafka integration. |
| `KAFKA_PASSWORD` | | Leave ALL of the following `KAFKA_` fields empty to skip the Kafka integration. |
| `KAFKA_TOPIC` | | |
| `KAFKA_SEED_BROKERS` | | A comma separated list of Kafka brokers, e.g. first-kafka-broker.domain.com:9092,second-kafka-broker.domain.com:9092 |
| `KAFKA_TOPICS_CONFIG` | | A Json configuration for a Kafka Topic to publish updates to. Leave empty to skip the Kafka integration. See below for details and an example. |
| `KAFKA_GROUP_ID_OVERRIDE` | | Override the kafka group id for local development to avoid creating lots of consumer groups. If unset, derived from local IP address so each k8s pod gets their own group. |
| | | |
| `AUTH_OIDC_KEY_SET_URL` | | URL to the [OpenID Connect Keyset][openid] for validating JWTs. See [authentication](#authentication) for more details. |
Expand Down Expand Up @@ -90,6 +87,9 @@ the [`local-config.yaml`][config] can be used to set the variables.
| `REPOSITORY_KEY_SEPARATOR` | `.` | Single character used to separate repository name from repository type. repository name and repository type must not contain separator. |
| | | |
| `ALLOWED_FILE_CATEGORIES` | | List of allowed keys for the filecategory field in repositories. Parsed as a json array, example value: `["key1","key2"]`. All keys not in this list are rejected on writes, and silently dropped when reading. |
| | | |
| `REDIS_URL` | | Url to an optional Redis instance to use as a shared cache. Will use in-memory cache if left blank |
| `REDIS_PASSWORD` | | Password for the Redis instance. Can be read from Vault via `VAULT_SECRETS_CONFIG` |

## Datastore

Expand Down Expand Up @@ -187,6 +187,41 @@ _If you are a client subscribing to our Kafka update notifications, and you want
state following an update notification, you must compare the commit hash and timestamp to see if you got the
correct version. If not, wait a bit and try again, you landed on an instance that isn't consistent yet._

### Kafka configuration

If you wish to use a Kafka topic, set the environment variable `KAFKA_TOPICS_CONFIG` to a JSON document
as follows (displayed in prettyprinted form for readability):

```
{
"metadata-change-events": {
"topic": "metadata-change-events",
"brokers": [
"kafka-seed-broker1.example.com:9092",
"kafka-seed-broker2.example.com:9092"
],
"username": "<username used to connect>",
"passwordEnvVar": "METADATA_CHANGE_EVENTS_PASSWORD",
"authType": "PLAIN"
}
}
```

This assumes of course that the password is provided in the specified environment variable. On Localhost, you
can simply set "password" in the JSON.

AuthType sets the SASL authentication method, possible values are `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`.

Note: You can use the Vault integration configuration to read the password from Vault by including it in
`VAULT_SECRETS_CONFIG`, similar to:

```
[...]
"some/vault/path": [
{"vaultKey": "METADATA_CHANGE_EVENTS_PASSWORD"}
],
```

## architecture

![software architecture](docs/architecture-export.png)
Expand Down Expand Up @@ -241,8 +276,7 @@ Clear the test cache:
### Goland terminal configuration

Goland has the annoying habit of limiting line width on the output terminal to 80 characters no matter how wide the
window is.
You can fix this. Menu: Help -> Find Action... -> search for "Registry"
window is. You can fix this. Menu: Help -> Find Action... -> search for "Registry"

Uncheck `go.run.processes.with.pty`.

Expand Down
16 changes: 0 additions & 16 deletions internal/repository/config/accessors.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -60,22 +60,6 @@ func (c *CustomConfigImpl) GitCommitterEmail() string {
return c.VGitCommitterEmail
}

func (c *CustomConfigImpl) KafkaUsername() string {
return c.VKafkaUsername
}

func (c *CustomConfigImpl) KafkaPassword() string {
return c.VKafkaPassword
}

func (c *CustomConfigImpl) KafkaTopic() string {
return c.VKafkaTopic
}

func (c *CustomConfigImpl) KafkaSeedBrokers() string {
return c.VKafkaSeedBrokers
}

func (c *CustomConfigImpl) AuthOidcKeySetUrl() string {
return c.VAuthOidcKeySetUrl
}
Expand Down
28 changes: 0 additions & 28 deletions internal/repository/config/config.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -99,34 +99,6 @@ var CustomConfigItems = []auconfigapi.ConfigItem{
Description: "email address to use for git commits",
Validate: auconfigenv.ObtainNotEmptyValidator(),
},
{
Key: config.KeyKafkaUsername,
EnvName: config.KeyKafkaUsername,
Default: "",
Description: "optional: kafka username (needed to send kafka notifications), leaving this or any of the other *KAFKA* fields empty will switch off all Kafka functionality",
Validate: auconfigapi.ConfigNeedsNoValidation,
},
{
Key: config.KeyKafkaPassword,
EnvName: config.KeyKafkaPassword,
Default: "",
Description: "optional: kafka password (needed to send kafka notifications), leaving this or any of the other *KAFKA* fields empty will switch off all Kafka functionality",
Validate: auconfigapi.ConfigNeedsNoValidation,
},
{
Key: config.KeyKafkaTopic,
EnvName: config.KeyKafkaTopic,
Default: "",
Description: "optional: kafka topic (needed to send kafka notifications), leaving this or any of the other *KAFKA* fields empty will switch off all Kafka functionality",
Validate: auconfigenv.ObtainPatternValidator("^(|[a-z0-9-]+)$"),
},
{
Key: config.KeyKafkaSeedBrokers,
EnvName: config.KeyKafkaSeedBrokers,
Default: "",
Description: "optional: comma separated list of kafka seed broker URLs (needed to send kafka notifications), leaving this or any of the other *KAFKA* fields empty will switch off all Kafka functionality",
Validate: auconfigenv.ObtainPatternValidator("^(|([a-z0-9-]+.[a-z0-9-]+.[a-z]{2,3}:9092)(,[a-z0-9-]+.[a-z0-9-]+.[a-z]{2,3}:9092)*)$"),
},
{
Key: config.KeyKafkaGroupIdOverride,
EnvName: config.KeyKafkaGroupIdOverride,
Expand Down
8 changes: 0 additions & 8 deletions internal/repository/config/plumbing.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,6 @@ type CustomConfigImpl struct {
VBitbucketReviewerFallback string
VGitCommitterName string
VGitCommitterEmail string
VKafkaUsername string
VKafkaPassword string
VKafkaTopic string
VKafkaSeedBrokers string
VAuthOidcKeySetUrl string
VAuthOidcTokenAudience string
VAuthGroupWrite string
Expand Down Expand Up @@ -100,10 +96,6 @@ func (c *CustomConfigImpl) Obtain(getter func(key string) string) {
c.VBitbucketReviewerFallback = getter(config.KeyBitbucketReviewerFallback)
c.VGitCommitterName = getter(config.KeyGitCommitterName)
c.VGitCommitterEmail = getter(config.KeyGitCommitterEmail)
c.VKafkaUsername = getter(config.KeyKafkaUsername)
c.VKafkaPassword = getter(config.KeyKafkaPassword)
c.VKafkaTopic = getter(config.KeyKafkaTopic)
c.VKafkaSeedBrokers = getter(config.KeyKafkaSeedBrokers)
c.VKafkaGroupIdOverride = getter(config.KeyKafkaGroupIdOverride)
c.VAuthOidcKeySetUrl = getter(config.KeyAuthOidcKeySetUrl)
c.VAuthOidcTokenAudience = getter(config.KeyAuthOidcTokenAudience)
Expand Down
13 changes: 5 additions & 8 deletions internal/repository/config/validation_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ func TestValidate_LotsOfErrors(t *testing.T) {
_, err := tstSetupCutAndLogRecorder(t, "invalid-config-values.yaml")

require.NotNil(t, err)
require.Contains(t, err.Error(), "some configuration values failed to validate or parse. There were 29 error(s). See details above")
require.Contains(t, err.Error(), "some configuration values failed to validate or parse. There were 27 error(s). See details above")

actualLog := goauzerolog.RecordedLogForTesting.String()

Expand All @@ -91,21 +91,18 @@ func TestValidate_LotsOfErrors(t *testing.T) {
expectedPart3 := "METRICS_PORT: value -12387192873invalid is not a valid integer"
require.Contains(t, actualLog, expectedPart3)

expectedPart4 := "failed to validate configuration field KAFKA_SEED_BROKERS: must match ^(|([a-z0-9-]+.[a-z0-9-]+.[a-z]{2,3}"
expectedPart4 := "failed to validate configuration field ALERT_TARGET_PREFIX: must match ^((http|https)://|)[a-z0-9-.]+.[a-z]{2,3}/$"
require.Contains(t, actualLog, expectedPart4)

expectedPart5 := "failed to validate configuration field ALERT_TARGET_PREFIX: must match ^((http|https)://|)[a-z0-9-.]+.[a-z]{2,3}/$"
expectedPart5 := "failed to validate configuration field ALERT_TARGET_SUFFIX: must match ^@[a-z0-9-]+.[a-z]{2,3}$"
require.Contains(t, actualLog, expectedPart5)

expectedPart6 := "failed to validate configuration field ALERT_TARGET_SUFFIX: must match ^@[a-z0-9-]+.[a-z]{2,3}$"
expectedPart6 := "failed to validate configuration field VAULT_ENABLED: value what is not a valid boolean value"
require.Contains(t, actualLog, expectedPart6)

expectedPart7 := "failed to validate configuration field VAULT_ENABLED: value what is not a valid boolean value"
expectedPart7 := "failed to validate configuration field VAULT_SECRETS_CONFIG: invalid character '}' after top-level value"
require.Contains(t, actualLog, expectedPart7)

expectedPart8 := "failed to validate configuration field VAULT_SECRETS_CONFIG: invalid character '}' after top-level value"
require.Contains(t, actualLog, expectedPart8)

require.Contains(t, actualLog, "failed to validate configuration field NOTIFICATION_CONSUMER_CONFIGS:")
require.Contains(t, actualLog, "Notification consumer config 'caseInvalidTypes' contains invalid type 'invalid'.")
require.Contains(t, actualLog, "Notification consumer config 'caseInvalidTypes' contains invalid type 'alsoInvalid'.")
Expand Down
20 changes: 0 additions & 20 deletions test/mock/configmock/configmock.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -127,26 +127,6 @@ func (c *MockConfig) GitCommitterEmail() string {
panic("implement me")
}

func (c *MockConfig) KafkaUsername() string {
//TODO implement me
panic("implement me")
}

func (c *MockConfig) KafkaPassword() string {
//TODO implement me
panic("implement me")
}

func (c *MockConfig) KafkaTopic() string {
//TODO implement me
panic("implement me")
}

func (c *MockConfig) KafkaSeedBrokers() string {
//TODO implement me
panic("implement me")
}

func (c *MockConfig) KafkaGroupIdOverride() string {
//TODO implement me
panic("implement me")
Expand Down
3 changes: 0 additions & 3 deletions test/resources/invalid-config-values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,6 @@ VAULT_SECRETS_CONFIG: '{}}'
UPDATE_JOB_INTERVAL_MINUTES: 26
UPDATE_JOB_TIMEOUT_SECONDS: true

KAFKA_USERNAME: 'WeirdÄÄÄÄ'
KAFKA_TOPIC: 'CRAZY/äöü'
KAFKA_SEED_BROKERS: 'external.url:443'
KAFKA_GROUP_ID_OVERRIDE: 'no banana, no spaces'

NOTIFICATION_CONSUMER_CONFIGS: >-
Expand Down
5 changes: 0 additions & 5 deletions test/resources/valid-config-unique.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,6 @@ BITBUCKET_REVIEWER_FALLBACK: username
GIT_COMMITTER_NAME: 'Body, Some'
GIT_COMMITTER_EMAIL: '[email protected]'

KAFKA_USERNAME: 'some-kafka-username'
KAFKA_PASSWORD: 'some-kafka-password'
KAFKA_TOPIC: 'some-kafka-topic'
KAFKA_SEED_BROKERS: 'first-kafka-broker.domain.com:9092,second-kafka-broker.domain.com:9092'

AUTH_OIDC_KEY_SET_URL: http://keyset
AUTH_OIDC_TOKEN_AUDIENCE: some-audience
AUTH_GROUP_WRITE: admin
Expand Down

0 comments on commit 5a69d44

Please sign in to comment.