Add additional logging for Azure repository stats

[saratvemulapalli]

Description

Adding additional logging for Azure repository testRequestStats to dump the requests ended up on the blob store.
These logs will only be printed when the tests fails.

The intention is to understand the number of calls received by Azure repo. The logger only dumps these when there is a failure in CI, shouldnot really bloat our runs.

Background: Ran this test 7k times and it successfully passed. See #9859 (comment). Looking at the errors we definitely need more logs for the number of calls being made by the SDK and what we really received.

Sample logs:

  1> [2024-01-30T11:00:23,635][INFO ][o.o.c.r.a.AllocationService] [node_s1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[index-no-merges][0]]]).
  1> [2024-01-30T11:00:23,745][INFO ][o.o.s.SnapshotsService   ] [node_s1] deleting snapshots [snapshot] from repository [u]
  1> [2024-01-30T11:00:23,866][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] GET /container?restype=container&comp=list&prefix=&delimiter=/&include=metadata
  1> [2024-01-30T11:00:23,980][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] GET /container?restype=container&comp=list&prefix=indices/&delimiter=/&include=metadata
  1> [2024-01-30T11:00:24,172][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] HEAD /container/index-0
  1> [2024-01-30T11:00:24,278][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] PUT /container/index-1
  1> [2024-01-30T11:00:24,384][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] PUT /container/index.latest
  1> [2024-01-30T11:00:24,570][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/index-0
  1> [2024-01-30T11:00:24,681][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/snap-ArzD5w73RHe2k00ZVn3C9A.dat
  1> [2024-01-30T11:00:24,683][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/index-0
  1> [2024-01-30T11:00:24,723][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/meta-ArzD5w73RHe2k00ZVn3C9A.dat
  1> [2024-01-30T11:00:24,833][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] GET /container?restype=container&comp=list&prefix=indices/BncAwVfxS_e51fLqhzjIqQ/
  1> [2024-01-30T11:00:24,945][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__TmGel7s5RsSmLDAWLVukwA
  1> [2024-01-30T11:00:24,945][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__-kcqFsxNShu1srufxxJZEw
  1> [2024-01-30T11:00:24,970][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2Findex-VhLeLRKxRUOFxVm7WEnyeA
  1> [2024-01-30T11:00:24,970][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__bc5nSe_HQpijW94eZ5yyNw
  1> [2024-01-30T11:00:24,971][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__UToqxFPEQwynYRQTJYMVKw
  1> [2024-01-30T11:00:24,971][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__K-gkZJjATreJxkUzv7oGqw
  1> [2024-01-30T11:00:24,972][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__QOPjyMOET0-FXKfEamZAhw
  1> [2024-01-30T11:00:24,972][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2Fsnap-ArzD5w73RHe2k00ZVn3C9A.dat
  1> [2024-01-30T11:00:24,973][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__RPDdKkaFS0Kr0Fmq3MvpGQ
  1> [2024-01-30T11:00:24,974][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__g_AlZPLhShKAsXlGI61e2A
  1> [2024-01-30T11:00:24,974][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__UaTIRPQ2S36AAwqmoXLk1w
  1> [2024-01-30T11:00:24,975][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__T_zGbpNDTyiz3bIDaLbHGQ
  1> [2024-01-30T11:00:24,975][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__1ue9RuigTnOPOnI4J5-GnQ
  1> [2024-01-30T11:00:24,975][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__2GANRHt8S1qi36v4QKSPzA
  1> [2024-01-30T11:00:24,976][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__vTRXd000QsOHkYkxHtljiA
  1> [2024-01-30T11:00:24,976][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__nLwJInPfSmCG39AvVULpWg
  1> [2024-01-30T11:00:24,982][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__P2t0ZLvgSV6JwBnmTRTy0A
  1> [2024-01-30T11:00:24,991][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__-Y9Dudt8QWuiyEo4rMXNTQ
  1> [2024-01-30T11:00:24,991][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2Fmeta-4CA8V40BKtO0Cfxc0NC3.dat
  1> [2024-01-30T11:00:24,992][INFO ][o.o.s.SnapshotsService   ] [node_s1] snapshots [snapshot/ArzD5w73RHe2k00ZVn3C9A] deleted

Related Issues

#9859

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 7679601: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

Compatibility status:

Checks if related components are compatible with change ed90c5d

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/sql.git]

[deshsidd]

Building failing? Please fix or retrigger the build

[saratvemulapalli]

Building failing? Please fix or retrigger the build

Yeah pretty weird, I haven't seen this. Looks like a network problem, not sure if its the tool server or jenkins CI.
Re-triggered with a push.

Build file '/var/jenkins/workspace/gradle-check/search/buildSrc/build.gradle' line: 279

* What went wrong:
A problem occurred configuring project ':build-tools'.
> Could not create task ':build-tools:spotlessJava'.
   > java.io.IOException: Failed to load eclipse jdt formatter: java.lang.RuntimeException: java.net.SocketTimeoutException: timeout

[github-actions]

✅ Gradle check result for 1f13238: SUCCESS

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (cfcb128) 71.43% compared to head (1f13238) 71.37%.

❗ Current head 1f13238 differs from pull request most recent head ed90c5d. Consider uploading reports for the commit ed90c5d to get more accurate results

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #12079      +/-   ##
============================================
- Coverage     71.43%   71.37%   -0.07%     
+ Complexity    59527    59499      -28     
============================================
  Files          4925     4925              
  Lines        279513   279513              
  Branches      40646    40643       -3     
============================================
- Hits         199674   199494     -180     
- Misses        63226    63473     +247     
+ Partials      16613    16546      -67     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

✅ Gradle check result for ed90c5d: SUCCESS