update grafana dashboard and exported metrics doc for watchable_panics_recovered_total

[DeeBi9]

What type of PR is this?

Update Grafana dashboard and exported metrics doc for watchable_panics_recovered_total

What this PR does / why we need it:

This PR updates the Grafana Dashboard to include the newly introduced metric watchable_panics_recovered_total, which tracks the total number of panics recovered in the Envoy Gateway system. It also updates the Exported Metrics Documentation to reflect the new metric and provide users with guidance on its use.

Which issue(s) this PR fixes:

#4728

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 66.81%. Comparing base (00ed0b8) to head (cdf548d).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5036      +/-   ##
==========================================
+ Coverage   66.75%   66.81%   +0.05%     
==========================================
  Files         209      209              
  Lines       32409    32409              
==========================================
+ Hits        21636    21655      +19     
+ Misses       9468     9455      -13     
+ Partials     1305     1299       -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[shawnh2]

Code overall looks good!

Can you run make -k gen-check to make the gencheck pass? Also can you add a screenshot image of this metric in grafana in the PR description ?

[DeeBi9]

This is the screenshot for the metric watchable_panics_recovered_total in the grafana dashboard , but the metrics appear on the dashboard does not display any data.

I have verified that the other metrics (like watchable_depth) are showing data, so the scraping is working.

I have checked the prometheus directly for the metrics watchable_panics_recovered_total but it doesn't appear there either.
I have followed the official documentation for deploying the Envoy Gateway and add-ons for observability. Is there a specific configuration or condition required for this metric to work ?

Thanks

[DeeBi9]

I also ran the make -k gen-check command locally as requested. Initially, I encountered an error similar to the one the codecov bot reported. This led to the addition of some test files, these are :

which I committed after seeing that two directories were modified. Upon committing the changes, I re-ran the command make -k gen-check, and it successfully completed without issues. However, I noticed that the section describing the watchable_panics_recovered_total metric in the docs was removed during this process, but the test ran smoothly afterward. Could you please clarify why this section was removed during the test execution? Let me know if there’s something specific I missed or if this is expected behavior.

Thanks