company-metrics-consumer

company-metrics-consumer is responsible for transforming company metrics data from the company-metrics kafka as part of chips and chs data sync

Development

Common commands used for development and running locally can be found in the Makefile, each make target has a description which can be listed by running make help

Target               Description
------               -----------
all                  Calls methods required to build a locally runnable version, typically the build target
build                Pull down any dependencies and compile code into an executable if required
clean                Reset repo to pre-build state (i.e. a clean checkout state)
deps                 Install dependencies
docker/kafka         Run kafka and create topics within docker
docker/kafka-create-topics Create kafka topics within docker
docker/kafka-start   Run kafka within docker
docker/kafka-stop    Stop kafka within docker
package              Create a single versioned deployable package (i.e. jar, zip, tar, etc.). May be dependent on the build target being run before package
sonar                Run sonar scan
test                 Run all test-* targets (convenience method for developers)
test-integration     Run integration tests
test-unit            Run unit tests

Running kafka locally

From root folder of this project run docker-compose up -d

Once containers up, run docker-compose exec kafka bash to enter kafka bash to create topics

Create kafka topics locally

kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic company-metrics

kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic company-metrics-retry

kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic company-metrics-error

Create kafka topics locally

kafka-topics.sh --list --zookeeper zookeeper:2181

Produce kafka test messages locally

kafka-console-producer.sh --topic delta-topic --broker-list localhost:9092

Terraform ECS

What does this code do?

The code present in this repository is used to define and deploy a dockerised container in AWS ECS. This is done by calling a module from terraform-modules. Application specific attributes are injected and the service is then deployed using Terraform via the CICD platform 'Concourse'.

Application specific attributes	Value	Description
ECS Cluster	data-sync	ECS cluster (stack) the service belongs to
Load balancer	non required as batch service	The load balancer that sits in front of the service
Concourse pipeline	Pipeline link Pipeline code	Concourse pipeline link in shared services

Contributing

• Please refer to the ECS Development and Infrastructure Documentation for detailed information on the infrastructure being deployed.

Testing

• Ensure the terraform runner local plan executes without issues. For information on terraform runners please see the Terraform Runner Quickstart guide.
• If you encounter any issues or have questions, reach out to the team on the #platform slack channel.

Vault Configuration Updates

• Any secrets required for this service will be stored in Vault. For any updates to the Vault configuration, please consult with the #platform team and submit a workflow request.

Useful Links

• ECS service config dev repository
• ECS service config production repository