Skip to content
This repository has been archived by the owner on Sep 2, 2021. It is now read-only.

adding support for custom templates #22

Merged
merged 4 commits into from
Oct 26, 2020
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,15 +176,23 @@ e46bfc69b893b659c4db713ba2cc84f1 GET /some/other/api

### Different output formats

By default Replay Zero generates Karate `*.feature` files and outputs them to the directory Replay Zero was started in. But the `--output` flag allows you to specify the format you'd like your tests to be generated in. The created files will always following the naming format `replay_scenarios_{N}.{extension}`. Currently the supported test formats are
By default Replay Zero generates Karate `*.feature` files and outputs them to the directory Replay Zero was started in. But the `--template` or `-t` flag allows you to specify the format you'd like your tests to be generated in. The created files will always following the naming format `replay_scenarios_{N}.{extension}`. Out of the box we support below test formats

* Karate (`*.feature`)
* Gatling (`*.scala`)

Specify the output as a lowercase input to the flag:

```sh
replay-zero --output=gatling
replay-zero --template=gatling
```

You can also pass path to your own custom template (in case you dont want to use karate or gatling) to the same paramater and `--extension` or `-e` to pass extentsion of the output.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I give this PR a green light after 2 quick additions

  1. Can we mention here that the templates are following the format from the text/template package in the Go stdlib? So readers have more guidance on where to start
  2. While we're at it, let's also let them know that we support all template functions provided by Sprig (ex. "You can even extend your template's functionality with string manipulation, math operators, and more from the Sprig library"), so they don't end up re-inventing like the wheel like I originally did when this project was started!

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!


E.g.

```sh
replay-zero --template=/path/to/your/custom-template --extension=java
```

## Roadmap
Expand Down
2 changes: 2 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,13 @@ require (
github.com/Masterminds/sprig v2.22.0+incompatible
github.com/aws/aws-sdk-go v1.29.25
github.com/go-test/deep v1.0.5
github.com/gobuffalo/here v0.6.2 // indirect
github.com/google/uuid v1.1.1 // indirect
github.com/huandu/xstrings v1.3.2 // indirect
github.com/imdario/mergo v0.3.11 // indirect
github.com/jmespath/go-jmespath v0.3.0 // indirect
github.com/kylelemons/godebug v1.1.0
github.com/markbates/pkger v0.17.1
github.com/mitchellh/copystructure v1.0.0 // indirect
github.com/nu7hatch/gouuid v0.0.0-20131221200532-179d4d0c4d8d
github.com/spf13/pflag v1.0.5
Expand Down
19 changes: 19 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,15 @@ github.com/aws/aws-sdk-go v1.29.25 h1:03yt6K6vCxfxFHT7mXGqFeISsNDE27LO2u+5AQBnTz
github.com/aws/aws-sdk-go v1.29.25/go.mod h1:1KvfttTE3SPKMpo8g2c6jL3ZKfXtFvKscTgahTma5Xg=
github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/go-sql-driver/mysql v1.5.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg=
github.com/go-test/deep v1.0.5 h1:AKODKU3pDH1RzZzm6YZu77YWtEAq6uh1rLIAQlay2qc=
github.com/go-test/deep v1.0.5/go.mod h1:QV8Hv/iy04NyLBxAdO9njL0iVPN1S4d/A3NVv1V36o8=
github.com/gobuffalo/here v0.6.0 h1:hYrd0a6gDmWxBM4TnrGw8mQg24iSVoIkHEk7FodQcBI=
github.com/gobuffalo/here v0.6.0/go.mod h1:wAG085dHOYqUpf+Ap+WOdrPTp5IYcDAs/x7PLa8Y5fM=
github.com/gobuffalo/here v0.6.2 h1:ZtCqC7F9ou3moLbYfHM1Tj+gwHGgWhjyRjVjsir9BE0=
github.com/gobuffalo/here v0.6.2/go.mod h1:D75Sq0p2BVHdgQu3vCRsXbg85rx943V19urJpqAVWjI=
github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/huandu/xstrings v1.3.2 h1:L18LIDzqlW6xN2rEkpdV8+oL/IXWJ1APd+vsdYy4Wdw=
Expand All @@ -21,8 +27,16 @@ github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af h1:pmfjZENx5i
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
github.com/jmespath/go-jmespath v0.3.0 h1:OS12ieG61fsCg5+qLJ+SsW9NicxNkg3b25OyT2yCeUc=
github.com/jmespath/go-jmespath v0.3.0/go.mod h1:9QtRXoHjLGCJ5IBSaohpXITPlowMeeYCZ7fLUTSywik=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.2.0 h1:s5hAObm+yFO5uHYt5dYjxi2rXrsnmRpJx4OYvIWUaQs=
github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/markbates/pkger v0.17.1 h1:/MKEtWqtc0mZvu9OinB9UzVN9iYCwLWuyUv4Bw+PCno=
github.com/markbates/pkger v0.17.1/go.mod h1:0JoVlrol20BSywW79rN3kdFFsE5xYM+rSCQDXbLhiuI=
github.com/mitchellh/copystructure v1.0.0 h1:Laisrj+bAB6b/yJwB5Bt3ITZhGJdqmxquMKeZ+mmkFQ=
github.com/mitchellh/copystructure v1.0.0/go.mod h1:SNtv71yrdKgLRyLFxmLdkAbkKEFWgYaq1OVrnRcwhnw=
github.com/mitchellh/reflectwalk v1.0.0 h1:9D+8oIskB4VJBN5SFlmc27fSlIBZaov1Wpk/IfikLNY=
Expand Down Expand Up @@ -51,7 +65,12 @@ golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.7/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
64 changes: 51 additions & 13 deletions main.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@ import (
"strings"
"syscall"

"github.com/markbates/pkger"
flag "github.com/spf13/pflag"

"github.com/intuit/replay-zero/templates"
"github.com/ztrue/shutdown"
)

Expand All @@ -31,7 +31,8 @@ var (
listenPort int
defaultTargetPort int
batchSize int
output string
template string
extension string
debug bool
streamRoleArn string
streamName string
Expand Down Expand Up @@ -81,10 +82,11 @@ func readFlags() {
}
flag.BoolVarP(&flags.version, "version", "V", false, "Print version info and exit")
flag.IntVarP(&flags.listenPort, "listen-port", "l", 9000, "The port the Replay Zero proxy will listen on")
flag.IntVarP(&flags.defaultTargetPort, "target-port", "t", 8080, "The port the Replay Zero proxy will forward to on localhost")
flag.IntVarP(&flags.defaultTargetPort, "target-port", "p", 8080, "The port the Replay Zero proxy will forward to on localhost")
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to update doc about this, if this is mentioned somewhere

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See latest review comment for more on docs

flag.BoolVar(&flags.debug, "debug", false, "Set logging to also print debug messages")
flag.IntVarP(&flags.batchSize, "batch-size", "b", 1, "Buffer events before writing out to a file")
flag.StringVarP(&flags.output, "output", "o", "karate", "Either [karate] or [gatling]")
flag.StringVarP(&flags.template, "template", "t", "karate", "Either [karate] or [gatling] or [path/to/custom/template]")
flag.StringVarP(&flags.extension, "extension", "e", "", "For custom output template")
Comment on lines +88 to +89
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will update documentation and add details about them after change is finalised

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See latest review comment for more on docs

flag.StringVarP(&flags.streamName, "stream-name", "s", "", "AWS Kinesis Stream name (streaming mode only)")
flag.StringVarP(&flags.streamRoleArn, "stream-role-arn", "r", "", "AWS Kinesis Stream ARN (streaming mode only)")
flag.Parse()
Expand All @@ -100,25 +102,61 @@ func readFlags() {
} else if flags.batchSize > 1 {
log.Printf("Using fixed batch size of %d\n", flags.batchSize)
}

// check if template exist at the provided path
// TODO: add validation for correctness of template
if !(flags.template == "karate" || flags.template == "gatling") {
_, err := ioutil.ReadFile(flags.template)
if err != nil {
log.Printf("Failed to load template")
panic(err)
}
if flags.extension == "" {
log.Fatal("For custom template, output extension is expected to be passed using --extension or -e flag.")
os.Exit(0)
}
}
}

// get embeded template file content
func getPkgTemplate(filePath string) string {
f, err := pkger.Open(filePath)
if err != nil {
panic(err)
}
defer f.Close()
info, err := f.Stat()
if err != nil {
panic(err)
}
b := make([]byte, info.Size())
content, err := f.Read(b)
if err != nil {
panic(err)
}
return string(b[:content])
}

func getFormat(output string) outputFormat {
switch output {
func getFormat(template string, extension string) outputFormat {
switch template {
case "karate":
return outputFormat{
template: templates.KarateBase,
template: getPkgTemplate("/templates/karate_default.template"),
extension: "feature",
}
case "gatling":
return outputFormat{
template: templates.GatlingBase,
template: getPkgTemplate("/templates/gatling_default.template"),
extension: "scala",
}
default:
log.Printf("Unknown output '%s', defaulting to karate", output)
dat, err := ioutil.ReadFile(template)
if err != nil {
panic(err)
}
return outputFormat{
template: templates.KarateBase,
extension: "feature",
template: string(dat),
extension: extension,
}
}
}
Expand Down Expand Up @@ -207,8 +245,8 @@ func main() {
log.Println("Running ONLINE, sending recorded events to Kinesis")
h = getOnlineHandler(flags.streamName, flags.streamRoleArn)
} else {
log.Printf("Running OFFLINE, writing out events to %s files\n", flags.output)
h = getOfflineHandler(flags.output)
log.Printf("Running OFFLINE, writing out events to %s files\n", flags.template)
h = getOfflineHandler(flags.template, flags.extension)
}

shutdown.Add(func() {
Expand Down
4 changes: 2 additions & 2 deletions offline.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,9 @@ type offlineHandler struct {
templateFuncMap template.FuncMap
}

func getOfflineHandler(output string) eventHandler {
func getOfflineHandler(template string, extension string) eventHandler {
return &offlineHandler{
format: getFormat(output),
format: getFormat(template, extension),
defaultBatchSize: flags.batchSize,
currentBatchSize: flags.batchSize,
writerFactory: getFileWriter,
Expand Down
66 changes: 66 additions & 0 deletions templates/gatling_default.template
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
package com.yourcompany.perf.yourservice
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wtait1 Do we want to continue to ship with the gatling template? We already have it, so may as well keep it?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Celeo My suggestion to this is, that we should stop shipping .go files as default supported template and convert them to .template. I did this to demonstrate the same. And a user can have more templates like this. Let me know if i make sense.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shubhamarora yes I think moving to .template files is great! In fact, what I'm thinking is that the Karate template should be moved over now as well. Then all test generation is being driven by a .template file, but the "default" formats use .template files that are shipped with the binary, and custom templates are provided via the new flag.

For packaging the default templates into the binary, there are several tools that do this. pkger seems like a good choice (go-bindata is no longer supported, and pkger is intended as a replacement for packr).
https://github.com/markbates/pkger

I've never personally worked with pkger before, but I'd be happy to help out / provide guidance as I think creating this common .template-based approach would be slick

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


// Generated by Replay Zero at {{ now }}

import io.gatling.core.Predef._
import io.gatling.core.structure.ScenarioBuilder
import io.gatling.http.Predef._
import io.gatling.http.protocol.HttpProtocolBuilder

class YourServiceBaseline extends Simulation {

val TP99_RESPONSE_TIME = 1600
val TP90_RESPONSE_TIME = 800
val TP50_RESPONSE_TIME = 600
val AVAILABILITY_RATE = 99

val testDuration: Integer = java.lang.Integer.getInteger("duration", 10)
val rampDuration: Integer = java.lang.Integer.getInteger("ramp", 2)
require(rampDuration > 0, "Ramp duration must be a natural number!")

val steadyStateDuration: Integer = testDuration.intValue() - rampDuration.intValue()
require(steadyStateDuration > 0, "Total test duration must be greater than ramp duration!")

val initialTps: Int = java.lang.Integer.getInteger("initial", 5)
val steadyStateTps: Int = java.lang.Integer.getInteger("steady", 100)
val host: String = System.getProperty("host", "TODO: put your service's base URL here")
val httpProtocol: HttpProtocolBuilder = http.baseURL(host)

{{ range $index, $event := . -}}
val scenario_{{$index}}: ScenarioBuilder = scenario("scenario_{{$index}}")
.exec(http("http_{{$index}}"))
.{{lower $event.HTTPMethod}}("{{$event.Endpoint}}")
{{- if gt (len $event.ReqHeaders) 0}}
.headers(
{{- range $h_index, $header := $event.ReqHeaders}}
{{- if $h_index -}},{{end}}
"{{$header.Name}}" = "{{$header.Value}}"
{{- end}}
)
{{- end}}
{{- if $event.ReqBody}}
.body(StringBody(
"""
{{ $event.ReqBody }}
"""
)){{- end }}

{{ end -}}
{{- println -}}
setUp(
{{- $toSkip := sub (len .) 1}}
{{- range $index, $event := . }}
scenario_{{$index}}.inject(rampUsersPerSec(initialTps) to steadyStateTps during (rampDuration minutes),
constantUsersPerSec(steadyStateTps) during (steadyStateDuration minutes)){{if ne $index $toSkip }},{{end}}
{{- end }}
)
.protocols(httpProtocol)
.assertions(
{{- range $index, $event := . -}}
{{- range $k, $v := dict "1" "99" "2" "90" "3" "50"}}
details("http_{{$index}}").responseTime.percentile{{$k}}.lessThan(TP{{$v}}_RESPONSE_TIME),
{{- end}}
{{- end}}
global.failedRequests.percent.lessThan(100 - AVAILABILITY_RATE)
)
}
34 changes: 34 additions & 0 deletions templates/karate_default.template
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Generated by Replay Zero at {{ now }}
Feature:

Background:
* url 'http://localhost:8080'
{{ range $index, $event := . }}
Scenario: test scenario {{.PairID}}
Given path '{{ $event.Endpoint }}'
{{/* add request headers if present */ -}}
{{ range $header := $event.ReqHeaders -}}
And header {{$header.Name}} = '{{$header.Value}}'
{{end }}
{{- /* add request body if present */ -}}
{{ if $event.ReqBody -}}
And request
"""
{{ $event.ReqBody }}
"""
{{- end }}

When method {{ $event.HTTPMethod }}
Then status {{ $event.ResponseCode }}
{{/* assert on response headers if present */ -}}
{{ range $header := $event.RespHeaders -}}
And match header {{$header.Name}} == '{{$header.Value}}'
{{end }}
{{- /* assert on response body if present */ -}}
{{ if $event.RespBody -}}
And match response ==
"""
{{ $event.RespBody }}
"""
{{- end }}
{{ end }}
28 changes: 28 additions & 0 deletions vendor/github.com/gobuffalo/here/.gitignore

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

45 changes: 45 additions & 0 deletions vendor/github.com/gobuffalo/here/.goreleaser.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading