Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite sgcollect in go #5783

Draft
wants to merge 21 commits into
base: main
Choose a base branch
from
Draft

Rewrite sgcollect in go #5783

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
6eb4b2d
WIP: rewrite sgcollect in go
markspolakovs Sep 20, 2022
4c3444b
basic auth
markspolakovs Sep 20, 2022
8a9e755
logs and configs
markspolakovs Sep 20, 2022
547fbfb
WIP redaction
markspolakovs Sep 20, 2022
76c1429
Working redaction
markspolakovs Sep 21, 2022
e716d7f
Refactoring: split files, combine interfaces into TaskEx
markspolakovs Sep 21, 2022
a72be08
Port over all OS tasks
markspolakovs Sep 21, 2022
63dfa34
Zip output, more tasks, redaction tweaks and tests
markspolakovs Sep 21, 2022
cfbe59f
Implement pprof and uploading
markspolakovs Sep 21, 2022
2a2f7d2
Remove OutputFile from SGCollectTask, keep on TaskEx
markspolakovs Sep 21, 2022
6f7465c
Refactor to allow for testing tasks
markspolakovs Sep 21, 2022
4685bd7
Remove sync_gateway imports to shrink binary
markspolakovs Sep 22, 2022
5fbf6fd
Add upload test
markspolakovs Sep 23, 2022
5af5fb4
Add an all-tasks test, and mark some as MayFail
markspolakovs Sep 23, 2022
6081d51
Suppress gosec + extra fuzz test for redaction
markspolakovs Sep 23, 2022
bad908c
Remove unused options
markspolakovs Sep 23, 2022
59448fa
Document command line flags
markspolakovs Sep 23, 2022
24e3161
Implement password remover
markspolakovs Sep 23, 2022
31d753d
Don't write headers for skipped tasks
markspolakovs Sep 23, 2022
afadc66
Remove old sgcollect
markspolakovs Sep 23, 2022
7f63ebe
mod tidy
markspolakovs Sep 23, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ require (
github.com/couchbase/go-couchbase v0.1.1
github.com/couchbase/gocb/v2 v2.5.3-0.20220803131303-46b466983d0f
github.com/couchbase/gocbcore/v10 v10.1.5-0.20220809160836-bf53e9527651
github.com/couchbase/gocbcore/v9 v9.1.8
github.com/couchbase/gomemcached v0.1.4
github.com/couchbase/goutils v0.1.2
github.com/couchbase/sg-bucket v0.0.0-20220916154817-791744ac79b7
Expand All @@ -35,18 +36,20 @@ require (
golang.org/x/exp v0.0.0-20220722155223-a9213eeb770e
golang.org/x/net v0.0.0-20220919232410-f2f64ebce3c1
golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8
gopkg.in/alecthomas/kingpin.v2 v2.2.6
gopkg.in/couchbase/gocb.v1 v1.6.7
gopkg.in/couchbase/gocbcore.v7 v7.1.18
gopkg.in/couchbaselabs/gocbconnstr.v1 v1.0.4
gopkg.in/square/go-jose.v2 v2.6.0
)

require (
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 // indirect
github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.1.2 // indirect
github.com/couchbase/blance v0.1.2 // indirect
github.com/couchbase/cbauth v0.1.1 // indirect
github.com/couchbase/gocbcore/v9 v9.1.8 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/go-ole/go-ole v1.2.6 // indirect
github.com/golang/protobuf v1.5.2 // indirect
Expand Down
16 changes: 3 additions & 13 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,11 @@ github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 h1:JYp7IbQjafoB+tBA3gMyHYHrpOtNuDiK/uB5uXxq5wM=
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d h1:UQZhZ2O0vMHr2cI+DC1Mbh0TJxzA3RcLoMsFw+aXw7E=
github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d/go.mod h1:rBZYJk541a8SKzHPHnH3zbiI+7dagKZ0cgpgrD7Fyho=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
Expand Down Expand Up @@ -79,20 +81,12 @@ github.com/couchbase/gomemcached v0.1.4 h1:5n5wmr4dBu+X7XteP8QHP5S9inK9MBjNpN9b7
github.com/couchbase/gomemcached v0.1.4/go.mod h1:mxliKQxOv84gQ0bJWbI+w9Wxdpt9HjDvgW9MjCym5Vo=
github.com/couchbase/goutils v0.1.2 h1:gWr8B6XNWPIhfalHNog3qQKfGiYyh4K4VhO3P2o9BCs=
github.com/couchbase/goutils v0.1.2/go.mod h1:h89Ek/tiOxxqjz30nPPlwZdQbdB8BwgnuBxeoUe/ViE=
github.com/couchbase/sg-bucket v0.0.0-20220725152948-e1112ff01a3d h1:cYrMXK8u0FQEKTes5PkRmbkao1EjESwH+6SHc7rDHsE=
github.com/couchbase/sg-bucket v0.0.0-20220725152948-e1112ff01a3d/go.mod h1:9XQoB1t+elPP+yEjHGOX3xcC3Z0/qDgOI7h/fc9XjlU=
github.com/couchbase/sg-bucket v0.0.0-20220824103435-aa28032bc2a3 h1:bPd/j0+YqKAOcekv0INIxjSiheB6Lle7wXAanGPb5Eo=
github.com/couchbase/sg-bucket v0.0.0-20220824103435-aa28032bc2a3/go.mod h1:9XQoB1t+elPP+yEjHGOX3xcC3Z0/qDgOI7h/fc9XjlU=
github.com/couchbase/sg-bucket v0.0.0-20220916154817-791744ac79b7 h1:0ahmhcMnxhExwEp6J+tEtHLw4kiUPU/Flb9HWmB9N2U=
github.com/couchbase/sg-bucket v0.0.0-20220916154817-791744ac79b7/go.mod h1:9XQoB1t+elPP+yEjHGOX3xcC3Z0/qDgOI7h/fc9XjlU=
github.com/couchbaselabs/go-fleecedelta v0.0.0-20200408160354-2ed3f45fde8f h1:al5DxXEBAUmINnP5dR950gL47424WzncuRpNdg0TWR0=
github.com/couchbaselabs/go-fleecedelta v0.0.0-20200408160354-2ed3f45fde8f/go.mod h1:daOs69VstinwoALl3wwWxjBf1nD4lIe3wwYhKHKDapY=
github.com/couchbaselabs/gocaves/client v0.0.0-20220223122017-22859b310bd2 h1:UlwJ2GWpZQAQCLHyO3xHKcqAjUUcX2w7FKpbxCIUQks=
github.com/couchbaselabs/gocaves/client v0.0.0-20220223122017-22859b310bd2/go.mod h1:AVekAZwIY2stsJOMWLAS/0uA/+qdp7pjO8EHnl61QkY=
github.com/couchbaselabs/walrus v0.0.0-20220726144228-c44d71d14a7a h1:hiVwAQnRHvo1oTYaOhNmxNg1KB0dTC9Htw/zDj0L0TU=
github.com/couchbaselabs/walrus v0.0.0-20220726144228-c44d71d14a7a/go.mod h1:C5TylJ1hRbYFgPnc/6gKUUPkLvkt7xDotgApqzd3dbg=
github.com/couchbaselabs/walrus v0.0.0-20220824104645-4bbd432c7128 h1:jvVNAyzrEyplEP+o43Y7zUYfxMD5m/6N2vf5kswEUZ8=
github.com/couchbaselabs/walrus v0.0.0-20220824104645-4bbd432c7128/go.mod h1:bg6u+qv616GUzyZng1swDEpgSn1k9FKpV+mxX7jAcAw=
github.com/couchbaselabs/walrus v0.0.0-20220916160453-6f7d5a152116 h1:/+J3rdBCFJieNnyQiFDSUBJnZD2D6Uh/1Dy4PXC+0WQ=
github.com/couchbaselabs/walrus v0.0.0-20220916160453-6f7d5a152116/go.mod h1:1Gy0YiYTNnuS4ThzYwKqz5X8q9qlCjAoqb/E1QbJdps=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
Expand Down Expand Up @@ -199,7 +193,6 @@ github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/gorilla/mux v1.8.0 h1:i40aqfkR1h2SlN9hojwV5ZA91wcXFOvkdNIeFDP5koI=
github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So=
github.com/gorilla/websocket v1.4.1 h1:q7AeDBpnBk8AogcD4DSag/Ukw/KV+YhzLj2bP5HvKCM=
github.com/gorilla/websocket v1.4.1/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/gorilla/websocket v1.4.2 h1:+/TMaTYc4QFitKJxsQ7Yye35DkWvkdLcvGKqM+x0Ufc=
github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
Expand Down Expand Up @@ -397,10 +390,6 @@ golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81R
golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220805013720-a33c5aa5df48 h1:N9Vc/rorQUDes6B9CNdIxAn5jODGj2wzfrei2x4wNj4=
golang.org/x/net v0.0.0-20220805013720-a33c5aa5df48/go.mod h1:YDH+HFinaLZZlnHAfSS6ZXJJ9M9t4Dl22yv3iI2vPwk=
golang.org/x/net v0.0.0-20220822230855-b0a4917ee28c h1:JVAXQ10yGGVbSyoer5VILysz6YKjdNT2bsvlayjqhes=
golang.org/x/net v0.0.0-20220822230855-b0a4917ee28c/go.mod h1:YDH+HFinaLZZlnHAfSS6ZXJJ9M9t4Dl22yv3iI2vPwk=
golang.org/x/net v0.0.0-20220919232410-f2f64ebce3c1 h1:TWZxd/th7FbRSMret2MVQdlI8uT49QEtwZdvJrxjEHU=
golang.org/x/net v0.0.0-20220919232410-f2f64ebce3c1/go.mod h1:YDH+HFinaLZZlnHAfSS6ZXJJ9M9t4Dl22yv3iI2vPwk=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
Expand Down Expand Up @@ -596,6 +585,7 @@ google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlba
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.26.0 h1:bxAC2xTBsZGibn2RTntX0oH50xLsqy1OxA9tTL3p/lk=
google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
gopkg.in/alecthomas/kingpin.v2 v2.2.6 h1:jMFz6MfLP0/4fUyZle81rXUoxOBFi19VUFKVDOQfozc=
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
Expand Down
Loading