Sauron, the all seeing eye! It is a service to generate automated reports and track migrations, changes and dependency versions for backend services also report on known CVE and security issues. A detailed description can be found in the internal RFC document.
Sauron Service is segregated into a few components that are described below:
-
Sauron Core: Sauron common library to be used by all sauron plugins. More details here
-
Sauron Service: Sauron entrypoint controller described in more details in this README
-
Sauron Plugin System: Sauron has an embedded plugin support that allows anyone to introduce its own logic without the need to rebuild, regenerate and stop/restart Sauron Service. More details in further sections.
-
Elasticsearch Cluster: Sauron uses Elasticsearch as a data storage. It is currently deployed in AWS and can be accessed here.
-
Kibana: Elasticsearch data can be explored using the built-in Kibana instance that can be accessed here.
-
Dependencytrack: Sauron includes an instance of Dependencytrack, a platform that allows organizations to identify and reduce risk from the use of third-party and open source components.
Sauron can be deployed using Docker and Docker-Compose. It provides a Dockerfile that can be built using the following commands:
Build Sauron project using maven and ship it to a docker image:
make
Start Sauron stack and load the local plugin repository
docker-compose -f docker-compose.yml --compatibility up
This command will start three components:
- elasticsearch: http://localhost:9200
- kibana: http://localhost:5601
- sauron-service: http://localhost:8080
Note: Since Sauron needs your maven, gradle, pip and nodejs configuration, the docker-compose.yml creates a volume
for each configuration folder/file. If there is no configuration already created, please create one before running
the command above. For more details please refer to
docker-compose.yaml > Volumes
In order to run Sauron with you predefined configuration using docker, use the command below:
docker run \
-e SPRING_CONFIG_LOCATION="/sauron/config/sauron-service.yml" \
-e M2_HOME="/usr/share/maven" \
-e SPRING_PROFILES_INCLUDE="local" \
-e SPRING_CLOUD_CONFIG_ENABLED="false" \
--mount type=bind,source=${PWD}/sauron-service/docker/config/sauron-service.yml,destination=/sauron/config/sauron-service.yml,readonly \
--mount type=bind,source=${PWD}/sauron-service/plugins,destination=/sauron/plugins \
--mount type=bind,source=${HOME}/.m2,destination=/root/.m2 \
--mount type=bind,source=${HOME}/.gradle,destination=/root/.gradle \
--mount type=bind,source=${HOME}/.pip,destination=/root/.pip \
--mount type=bind,source=${HOME}/.npmrc,destination=/root/.npmrc \
--mount type=bind,source=${HOME}/.ssh,destination=/root/.ssh,readonly \
--name=sauron \
-p 8080:8080 \
ghcr.io/freenowtech/sauron/sauron-service:latestIf you need to use a specific version, please refer to Sauron Packages
Sauron configuration can be set via application.properties file. The file path can be provided using the environment
variable:
- SPRING_CONFIG_LOCATION: it must contain the path pointing to the local file
e.g.
/path/to/config/my-properties.propertiesor/path/to/config/my-properties.yaml
Sauron supports Spring Cloud Config Server as a configuration provider. In order to set Config Server url, please use the environment variable below:
- SPRING_CLOUD_CONFIG_URI: it must contain the URL pointing to the remote repository
e.g.
https://my-repository.com/my-config
For a Sauron configuration file example, please refer to sauron-service.yml
Before start using Sauron, it is import to define the index template that will be used by Elasticsearch to create Sauron's index. It can be done by running the command line below:
elasticsearch/sauron-template.sh
elasticsearch/dependencies-template.sh
This template increases the number of fields, since usually the amount of dependencies and thus the amount of fields is huge, and some other minor optimizations.
Sauron Service provides a REST api that allows one to trigger a new build. The detailed parameters can be found in its swagger documentation. Once a new build has been triggered, Sauron's pipeline will run applying all plugins pre-configured. The output will be stored in elasticsearch and can be queried afterwards using the Kibana Installation.
For more details, check in the next section how the plugin system works.
A build request example can be found below:
curl --verbose --location --request POST 'http://localhost:8080/api/v1/build' \
--header 'Content-Type: application/json' \
--data-raw '{
"serviceName": "MyService",
"repositoryUrl": "https://github.com/gazgeek/springboot-helloworld.git",
"commitId": "41c7823dddbef43680a0726ccea0631519b9d3c1",
"buildId": "2b8caa6c-8b55-4b57-b654-5a00d519f409",
"owner": "Sauron",
"eventTime": 1586962717770,
"rollback": false,
"returnCode": 0,
"environment": "production",
"release": "0.0.1",
"user": "sauron-user",
"dockerImage": "helloworld:0.0.1",
"platform": "K8S"
}'As mentioned before, Sauron stack uses Elasticsearch + Kibana to respectively store and visualize data. Once the Sauron build is done, the information will be available and can be queried using Kibana.
In order to do that, a index pattern must be created in Kibana using this HOWTO.
Once it is done, you are able to use your data. Sauron provides a default Kibana Dashboard available here. Import it and voilà! Enjoy your data!
Sauron has an embedded plugin system that allows anyone to insert its own business logic to extract information during the building/deploy process. It uses the PF4J which is a plugin framework written in Java, and provides a nice interface to implement an integration in your service.
During the startup Sauron loads all available plugins, and updates them every 5 minutes using the pre-defined plugin repository (Local or Artifactory). For more details please refer to sauron-service.yml.
Prints the DataSet content to sysout.
Sanitizes the data before being processed by Sauron pipeline.
Checkout the project source code.
Extracts dependencies information in CycloneDX and insert in DataSet.
Publishes the dependencies to our internal Dependency Track instance.
Retrieves information from pom.xml file.
Stores the DataSet content into Elasticsearch.
Checks whether a service is using protoc, and the protoc wrapper.
Tells if your service has any logs on elasticsearch. You can configure indexes and time range to search for logs.
Allows Sauron to query Kubernetes API to retrieve the annotations and labels assigned to the resources, specified in the configuration, and stores the values into the DataSet.
Query Sonar API to retrieve your service related data, like Code Coverage, and stores into the DataSet.
Query Thanos API to retrieve your service related data(like RPM, Circuit Breaker), and stores into the DataSet.
Checks whether a service has or not a README.md file in its root folder.
Checks whether a service might be using Bcrypt to encode the passwords in the API
Query Jaeger API to extract tracing information
Cleanup Sauron workspace after each pipeline processing
Sauron provides a maven archetype in order to create a standard skeleton of a new plugin. To use that follow the steps below.
Checkout the sauron-plugin-archetype project:
$ git clone https://github.com/freenowtech/sauron.gitInstall the archetype locally and add it to the local archetype catalog:
$ cd sauron-plugin-archetype
$ mvn clean installGenerate the new plugin using the installed archetype:
$ mvn org.apache.maven.plugins:maven-archetype-plugin:3.1.2:generate
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] >>> maven-archetype-plugin:3.0.1:generate (default-cli) > generate-sources @ standalone-pom >>>
[INFO]
[INFO] <<< maven-archetype-plugin:3.0.1:generate (default-cli) < generate-sources @ standalone-pom <<<
[INFO]
[INFO]
[INFO] --- maven-archetype-plugin:3.0.1:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode
[INFO] No archetype defined. Using maven-archetype-quickstart (org.apache.maven.archetypes:maven-archetype-quickstart:1.0)
Choose archetype:
...
Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): : sauron-plugin-archetype
Choose archetype:
1: remote -> com.freenow.sauron:sauron-plugin-archetype (Sauron Plugin Archetype)
Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): : 1
[INFO] Using property: groupId = com.free-now.sauron.plugins
Define value for property 'artifactId': my-plugin
[INFO] Using property: version = 0.0.1-SNAPSHOT
[INFO] Using property: package = com.freenow.sauron.plugins
Define value for property 'className': MyPlugin
Confirm properties configuration:
groupId: com.free-now.sauron.plugins
artifactId: my-plugin
version: 0.0.1-SNAPSHOT
package: com.freenow.sauron.plugins
className: MyPlugin
Y: :
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: sauron-plugin-archetype:0.0.1-SNAPSHOT
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: com.free-now.sauron.plugins
[INFO] Parameter: artifactId, Value: my-plugin
[INFO] Parameter: version, Value: 0.0.1-SNAPSHOT
[INFO] Parameter: package, Value: com.freenow.sauron.plugins
[INFO] Parameter: packageInPathFormat, Value: com/freenow/sauron/plugins
[INFO] Parameter: package, Value: com.freenow.sauron.plugins
[INFO] Parameter: version, Value: 0.0.1-SNAPSHOT
[INFO] Parameter: groupId, Value: com.free-now.sauron.plugins
[INFO] Parameter: className, Value: MyPlugin
[INFO] Parameter: artifactId, Value: my-plugin
[INFO] Project created from Archetype in dir: /Users/sergio/Documents/sauron/my-plugin
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15.041 s
[INFO] Finished at: 2019-04-08T15:10:44+02:00
[INFO] ------------------------------------------------------------------------It will generate a new maven project that follows the structure below:
Fill in the MyPlugin.java class with your desired logic:
package com.freenow.sauron.plugins;
import com.freenow.sauron.model.DataSet;
import com.freenow.sauron.properties.PluginsConfigurationProperties;
import org.pf4j.Extension;
@Extension
public class MyPlugin implements SauronExtension
{
@Override
public DataSet apply(PluginsConfigurationProperties properties, DataSet input)
{
// PLUGIN LOGIC
return input;
}
}In order to provide extra configuration to your plugin, refer to CONFIGURATION SECTION.
After your configuration has been added, it will be available to be used by your plugin in
PluginsConfigurationProperties object.
See below an example of how you can use the configuration properties:
The following configuration provides an url to my-plugin plugin generated in step above:
sauron:
plugins:
my-plugin:
url: https://my-plugin.comSo in order to retrieve this configuration uses the following java code:
@Extension
public class MyPlugin implements SauronExtension
{
@Override
public DataSet apply(PluginsConfigurationProperties properties, DataSet input)
{
properties.getPluginConfigurationProperty("my-plugin", "url").ifPresent(url -> System.out.println(url) );
return input;
}
}Sauron Plugin Archetype provides a template of a README that must be filled to document what your plugin does, inputs, outputs and possible configuration.
Once the developing and testing process has been done, you can deploy a new version of your plugin, using the pre-defined plugin repository (Local or Artifactory). For more details please refer to sauron-service.yml.
The plugin reloading process runs every 5 minutes. To force a reloading use the /api/v1/reload method.
Check Swagger Documentation for more information.
Once your plugin has been deployed and loaded by Sauron Service you can use it in a pipeline which will be then applied to new deployments. Refer to Sauron Usage for detailed information.