Added check to events-to-s3 for label canary

[bshien]

Description

Added check to events-to-s3 for label canary

Issues Resolved

Part of opensearch-project/opensearch-metrics#105

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.47%. Comparing base (4a90494) to head (c07ade7).

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #40      +/-   ##
==========================================
+ Coverage   99.44%   99.47%   +0.02%     
==========================================
  Files          10       10              
  Lines         181      190       +9     
  Branches       25       26       +1     
==========================================
+ Hits          180      189       +9     
  Misses          1        1              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[peterzhuamazon]

Not really sure what you are trying to do.

Are you expecting a label called s3-data-lake-app-canary-label to be added to an issue, then sending a metrics to cloudwatch ?

Thanks.

[peterzhuamazon]

Can you explain your use case here?

Thanks.

[bshien]

Sure, I'm trying to set up monitoring for this automation app running in the docker container in the metrics aws account so if the app goes down, we will get notified. There will be a lambda that will create and delete a label every 10 minutes, and once this app listens on that event, it will send a CW metric. There will be an alarm on that metric that triggers where there is no metric data(indicating that the automation app is down).

[peterzhuamazon]

In which repo you are doing this label creation/deletion?
Also is there any other ways to monitor this than touching a label?

Thanks.

[bshien]

I'm doing the label creation/deletion in opensearch-metrics. One sec, let me add a check for this.

Prudhvi and I discussed about it, there are a few alternative approaches like checking for logs but the canary seems best since it directly tests it in real time. For example, the issue with checking for logs is that certain times, like the weekend, have less events potentially causing the alarm to trigger where there is nothing wrong. The name I chose for this label is long and specific(s3-data-lake-app-canary-label) and it gets deleted immediately after creation, so it won't have much impact on anything else.

[peterzhuamazon]

Then you code in metrics repo should ensure that creation/deletion both return success before false positives being captured here.

Also I think you can force create label as well, so you might not need to delete the label anyway.
Just create a label with -Force param or similar would be enough, since you only want canary to add data to cloudwatch every X min to ensure there is no alarm.

[prudhvigodithi]

Any small and simplified name please?

[bshien]

I think that a long name for the label might be better in case someone adds this label randomly

[peterzhuamazon]

Or you can just use a random hash if you just want to test the addition and deletion.

[peterzhuamazon]

Please rebase and update your version to 0.2.2.

Thanks.