GPII-3394: Add couchdb support

[mrtyler]

This adds couchdb support to stackdriver-agent.

This PR takes the Google-provided CouchDB config and:

• Adds shell stackdriver-agent shell wrapper.
• Changes to CouchDB 2.x metrics names.
• Exports metrics in custom namespace since they no longer match predefined metrics names.

Next steps include:

• Review a few CouchDB monitoring guides I found for other metrics to track
• Add CI for our fork of stackdriver-agent including publishing Docker images (I'm inclined to use Gitlab since we already have it configured to publish to Docker Hub; however I'm open to using CircleCI if someone can convince me it will be quick and painless :))
• Integrate with Helm and gpii-infra
• Resolve some other tactical issues (_local URLs, disabling redundant collection of system metrics)

I will PR these changes to upstream after review here.

[natarajaya]

LGTM!

[stepanstipl]

LGTM.

On a more general note this seems to be quite a tedious process, having to manually configure every single metric. Might something like https://github.com/gesellix/couchdb-prometheus-exporter together with https://github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/prometheus-to-sd be more elegant and maintainable solution?

[amatas]

LGTM

+1 to simplifing the way to manage the configuration, but if a lot of time is required for that change I think that this PR is ok for now and leave the configuration management for the future.

[mrtyler]

@stepanstipl

this seems to be quite a tedious process, having to manually configure every single metric

Agreed. :)

There is a (small, for now) upside: Stackdriver is pay-per-byte-of-metric-traffic so logging exactly what we want may save us a little money (I have no sense of scale for 150MiB free + $0.2580/MiB thereafter).

Might something like https://github.com/gesellix/couchdb-prometheus-exporter together with https://github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/prometheus-to-sd be more elegant and maintainable solution?

I like this idea. However, I'm also wary of investing more effort (and adding two more containers to the mix). I am happy to explore this area if the team thinks now is a good time to do so. If not, I'd like to ship this as-is, try it out, and start collecting metrics from couchdb with an eye toward better understanding the performance problem cited in GPII-3394.

What do y'all think?

[stepanstipl]

I like this idea. However, I'm also wary of investing more effort (and adding two more containers to the mix). I am happy to explore this area if the team thinks now is a good time to do so. If not, I'd like to ship this as-is, try it out, and start collecting metrics from couchdb with an eye toward better understanding the performance problem cited in GPII-3394.

Yep, I agree - basically whatever gets us there sooner with less effort at the moment :) and sounds like we have the metrics part done, so I agree we should focus on the original performance issue. I would only like to avoid investing significantly more time into this collecd/stackdriver agent setup without at least trying the other alternatives, should that be necessary.

[mrtyler]

After some investigation of couchdb-prometheus-exporter and prometheus-to-sd and a demo today, we decided to pursue that strategy instead: gpii-ops/gpii-infra#230

I proposed it to upstream anyway, in case it's useful to someone: wikiwi#4