DRAFT: Fix SnippetsFilter Functional test

[bjee19]

Running the pipeline, do not review yet.

Issue is that after applying the SnippetsFilter manifest, which contains the HTTPRoute and GRPCRoute, there was no check to see if the GRPCRoute is ready. We only had a check to make sure traffic flowed to the HTTPRoute. I believe this caused a race condition where the test could occasionally fail if NGF was still processing the GRPCRoute while it had already processed the HTTPRoute and moved forward in the test suite.

Closes: #3753

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.90%. Comparing base (235a4b2) to head (145aa73).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3751      +/-   ##
==========================================
- Coverage   86.90%   86.90%   -0.01%     
==========================================
  Files         128      128              
  Lines       16187    16201      +14     
  Branches       62       62              
==========================================
+ Hits        14067    14079      +12     
- Misses       1948     1949       +1     
- Partials      172      173       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

it was draft

[salonichf5]

omg hoping this is it, great find

[bjee19]

Adding flake attempts, not quite sure whats causing these 502 errors and traffic not flowing to the httproutes as seen here: https://github.com/nginx/nginx-gateway-fabric/actions/runs/17055138233/job/48355054192#step:17:729.

The "no live upstreams while connecting to upstream" is a common error we've seen in other tests so it could be related to that. This error also seems to be quite inconsistent, in the past 8 local runs and 12 (4 times per pipeline run, and i ran it 3 times) runs of a functional test in the pipeline it only occurred once.

[sjberman]

no live upstreams is indicative of a problem with the backend apps themselves, not nginx. It would be nice if we could diagnose why those might be having issues (like logs, for example).

[bjee19]

I guess in our case the backend apps are just nginxdemos/nginx-hello:plain-text which shouldn't really be facing issues, do you think its worth diagnosing them? Especially when its very infrequent? @sjberman

Actually, it looks like all the non-plus SF tests fail on the GRPC test case which should be solved by the waiting for apps to be ready check. But the Plus function test runs fail on the httproute not being able to send traffic. See https://github.com/nginx/nginx-gateway-fabric/actions/runs/17075769277/job/48417202770 for an example. I'll keep taking a look.

[sjberman]

@bjee19 Well there is clearly something going wrong here. If the apps themselves are having problems (or k8s is restarting the pods, or something else), it's causing our tests to be flaky, which is annoying. We do still have to ensure our infrastructure works. Can't ignore the potential for that issue just because it isn't a product issue. Or the other potential issue is that nginx has the wrong IP addresses for the endpoints. In that case, it could be our problem directly.

There are definitely cases where rerunning flaky tests is valid, but I also don't want us to hide issues.