Espv1 returns 502/Bad Gateway with code:13 randomly

[abdullah-retorio]

We have been using Espv1 for more than 4 years and it started returning 502/Bad Gateway randomly. The gateway works normal most of the time, but once a week, once a month for couple hours, it returns this.

We were using v1.30.0 for a long time, after this error we upgraded it to 1.59.0 but this didn't solve the error below.

{
    "code": 13,
    "message": "BAD_GATEWAY",
    "details": [
        {
            "@type": "[type.googleapis.com/google.rpc.DebugInfo](https://www.google.com/url?q=http://type.googleapis.com/google.rpc.DebugInfo&sa=D&source=buganizer&usg=AOvVaw3bPjFYIt2uLfMUT6OCRqFP)",
            "stackEntries": [],
            "detail": "application"
        }
    ]
}

This is the only log that we can see from cloud logging page. I censured some confidential parts in the log below.

[error] 5#5: *807 europe-west1-****.[cloudfunctions.net](https://www.google.com/url?q=http://cloudfunctions.net&sa=D&source=buganizer&usg=AOvVaw0CByUUXr-uvQLVuKgVdXbz) could not be resolved (110: Operation timed out), client: /*/*/*/*, server: , request: "POST /*-*-*-*- HTTP/1.1", host: "*-*-*-*-"

What you expected to happen:

We hadn't got this error before 3 months ago. We want to know what can be the reason and what should we do?

https://cloud.google.com/endpoints/docs/openapi/troubleshoot-response-errors I have checked this page but it doesn't help me because its not a complete Esp failure. It happens randomly and after couple hours, it disappears.

Here you can find some screenshots from cloud logging page. As you might have seen, there is no indicator in the logs and it took more than 4 hours in our production environment. It returns 502 for a couple of hours and then starts working normally. This happened multiple times, especially in the last couple of weeks.:

Error happening:

Error gone somehow:

[boguima]

Try to carry out a health validation

ex:
`
- args:
- '--http_port'
- '8081'
- '-z'
- healthz
- '--backend'
- '127.0.0.1:8080'
- '--service'
- >-
api.ins-cr-broker-list-be.endpoints.ins-aftersales-clusters-dev.cloud.goog
- '--rollout_strategy'
- managed
image: 'gcr.io/endpoints-release/endpoints-runtime:1'

`