enhance(logging): add flag to allow logging in ecs format, map custom fields to ecs fields

[KellyMerrick]

Add flag to allow logging in Elasticsearch Common Schema (ECS) format. Default will remain logrus.JSONFormatter.

Custom Vela fields will be nested under labels.vela, which allows separation from other non-Vela custom labels that may exist in one's logging stack.

server/cmd/vela-server/main.go

Lines 51 to 56 in a000fd0

	&cli.StringFlag{
	EnvVars: []string{"VELA_LOG_FORMATTER", "LOG_FORMATTER"},
	Name: "log-formatter",
	Usage: "set log formatter - options: (json|ecs)",
	Value: "json",
	},

server/cmd/vela-server/server.go

Lines 29 to 39 in a000fd0

	// set log formatter
	switch c.String("log-formatter") {
	case "json":
	// set logrus to log in JSON format
	logrus.SetFormatter(&logrus.JSONFormatter{})
	case "ecs":
	// set logrus to log in Elasticsearch Common Schema (ecs) format
	logrus.SetFormatter(&middleware.Formatter{
	DataKey: "labels.vela",
	})
	}

As noted below, a custom formatter, borrowing from ecslogrus, was created to handle Vela custom fields.

server/router/middleware/logger.go

Lines 23 to 25 in a000fd0

	// This file, in part, reproduces portions of
	// https://github.com/elastic/ecs-logging-go-logrus/blob/v1.0.0/formatter.go
	// to handle our custom fields in Format().

server/router/middleware/logger.go

Lines 145 to 192 in a000fd0

	// Format formats e as ECS-compliant JSON,
	// mapping our custom fields to ECS fields.
	func (f *Formatter) Format(e *logrus.Entry) ([]byte, error) {
	datahint := len(e.Data)
	if f.DataKey != "" {
	datahint = 2
	}
	data := make(logrus.Fields, datahint)
	if len(e.Data) > 0 {
	extraData := data
	if f.DataKey != "" {
	extraData = make(logrus.Fields, len(e.Data))
	}
	for k, v := range e.Data {
	switch k {
	case "ip":
	data["client.ip"] = v
	case "latency":
	data["event.duration"] = v
	case "method":
	data["http.request.method"] = v
	case "path":
	data["url.path"] = v
	case "status":
	data["http.response.status_code"] = v
	case "user-agent":
	data["user_agent.name"] = v
	case "version":
	data["user_agent.version"] = v
	default:
	extraData[k] = v
	}
	}
	if f.DataKey != "" && len(extraData) > 0 {
	data[f.DataKey] = extraData
	}
	}
	data["ecs.version"] = ecsVersion
	ecopy := *e
	ecopy.Data = data
	e = &ecopy
	jf := logrus.JSONFormatter{
	TimestampFormat: "2006-01-02T15:04:05.000Z0700",
	FieldMap: ecsFieldMap,
	}
	return jf.Format(e)
	}

default log:

{"build":"a build","ip":"123.4.5.6","latency":2303458,"level":"info","method":"GET","msg":"200 2.303458ms 123.4.5.6 GET /some/path","path":"/some/path","status":200,"time":"2023-09-26T10:03:27-05:00","user":"a user","user-agent":"user-agent field","version":"version field"}

ecs formatted log:

{"@timestamp":"2023-09-26T10:04:07.218-0500","client.ip":"123.4.5.6","ecs.version":"1.6.0","event.duration":2928917,"http.request.method":"GET","http.response.status_code":200,"labels.vela":{"build":"a build","user":"a user"},"log.level":"info","message":"200 2.928917ms 123.4.5.6 GET /some/path","url.path":"/some/path","user_agent.name":"user-agent field","user_agent.version":"version field"}

[codecov]

Codecov Report

Merging #971 (9eacb1a) into main (dff9633) will increase coverage by 0.10%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #971      +/-   ##
==========================================
+ Coverage   70.98%   71.09%   +0.10%     
==========================================
  Files         312      312              
  Lines       12934    12982      +48     
==========================================
+ Hits         9181     9229      +48     
  Misses       3288     3288              
  Partials      465      465              

Files	Coverage Δ
router/middleware/logger.go	97.50% <100.00%> (+1.66%)	⬆️