Edge function - get HTTP errors 520 / 546

[littleannie]

Hi there,

We have an application that makes an API call to one of our Supabase edge function.

Some of the calls fail, and we're getting HTTP errors 520 and 546.

These errors seem to be more frequent since the last 2 weeks (since around September 28th).

Does anyone know what these errors correspond to? Could it be related to Cloudflare, and if so, why?

Thanks in advance for any help!

[Eldynn]

We still experience this issue to this day. Is there anyone with experience to debug those cases?

[jordanful]

We're also seeing 546 responses, without further detail, when calling an edge function via a database webhook trigger, or when hitting a GET endpoint directly.

[jordanful]

Here's what we get with a direct GET request {"code":"WORKER_LIMIT","message":"Worker failed to respond due to an error or resource limit (please check logs)"}

[TSIA-SN]

@littleannie @Eldynn In your Supabase UI, would you please go to Edge Functions > {your edge function name} > Invocations and click on one of the errored calls, you should see a x_sb_edge_region header in the Response Metadata, could you post what region your servers are? On Discord, two of us having the issue are on us-east-1, and one is not having the issue but is on eu-west-3, so I figure it would be good to gather the server info as well in case that's related.

[laktek]

@littleannie I'm from the Supabase Edge Functions team and want to provide an update on the HTTP errors. We are currently migrating Edge Functions to use Supabase Edge Runtime instead of Deno Deploy. This should require no change to existing functions deployed to Supabase and mostly transparent.

You can check whether the function was served by Supabase Edge Runtime, if you check the response headers of a function call. It should have x-served-by set to supabase-edge-runtime. You should also see this in the edge function logs (in dashboard, Edge Functions > {Function Name> Logs) under served_by property. There's also a corresponding property (region) and a header (x-sb-edge-region), showing which region the request was served from.

The 520 errors you experienced last week should be mostly fixed. This was due to a misconfiguration on our end during the roll out in certain regions. If you still see 520 responses, please open a support request so we can investigate.

The 546 error status is returned when a resource limit of a function is reached. A resource limit can be CPU (50ms), memory (512mb) or wall clock time (ie. total time your function runs - 5mins). You should see the specific resource limit errors under the Logs section for specific edge function in Supabase dashboard.

Please open a support request if your functions are continuously experiencing resource limit errors (546s), we can help you figure out what could be the underlying cause.

[sansavision]

I'm also getting the 546 error while not violating any of these "A resource limit can be CPU (50ms), memory (512mb) or wall clock time (ie. total time your function runs - 5mins).". I am on eu-central1

[laktek]

@sansavision Have you checked Edge Functions logs to see if there are any errors? Usually it should log the reason why your function was terminated (which is when 546 error response is returned)

[vrijmetse]

No error logs on my side.

[vrijmetse]

I am also getting the 546 error, the funny thing is the function works as it should, making api call etc. However the response its still 546 and no error log whatsover.

[evelant]

@laktek Also running into this. The edge function logs do not get any entries when there is a failure. The only place any indication of a failure is available is through a logs query where there's little useful information, such as

select
  cast(t.timestamp as datetime) as timestamp,
  res.status_code,
  event_message,
  m.execution_time_ms,
  res.headers
from function_edge_logs as t
 cross join unnest(t.metadata) as m
 cross join unnest(m.response) as res
 cross join unnest(m.request) as req
where
  res.status_code > 400
order by
  timestamp desc
limit 1000

All I can tell is that edge function calls are failing with 502 or 546. There's no information about these failures in the edge function logs on the dashboard. For some of them I see a log from the function starting and that's all, it just silently ends after that.

[foxspur]

As of 11/29 I'm facing this issue as well. It's very disappointing.

[evelant]

Error 546 means your function is using too much CPU time. Supabase support said that configurable CPU time limits are an upcoming feature.

I emailed Supabase support about the 502/504 random failures and apparently these random failures are known and expected at this point. They recommended adding retry logic to my app as a workaround. They said they're working on the issue but that it's complex.

[jordanful]

In case it is helpful: if there are any type errors in your function, it may run fine locally but bomb when you deploy it. Triple check your Typescript.

[liam-k]

Yes, or any other uncaught error too! I think a few of these reports here could be because people also had bad / nonexistent error handling in edge functions. It’s a bit cryptic that a 546 would be returned, but yeah, it can basically mean anything.

[Maor44]

as for Dec 2, I get this error too, and have no idea what I can do to fix it.

[jcmayoral]

Any solution for this? I got it too.

[camer0nluo]

I have the same issue ^

[YosukeImai]

I am also facing the same issue. Even with a simple function returning only query parameters, the issue still exists.

[masihsultani]

Are you doing Deno.serve?

[evelant]

I ended up ditching edge-functions for a simple Bun server running on fly.io. Too many random failures and pain with Deno on edge-functions. The process was pretty painless, basically these steps:

1. Install https://bun.sh/ and flyctl https://fly.io/docs/hands-on/install-flyctl/
2. Move your edge functions code to a new package/directory
3. Translate from Deno to normal TS. For me this was as simple as: add a package.json, add your deps and remove import_map.json, replace Deno.serve with an exported async function and Deno.env.get with Bun.env[...].
4. Add an index.ts with Bun.serve that calls your former-edge-function code appropriately based on const { pathname } = new URL(request.url)
5. flyctl launch to setup fly.toml, Dockerfile, and launch on fly.io
6. flyctl secrets add SUPABASE_DB_URL=your_db_url and any other secrets you may need such as SUPABASE_SERVICE_ROLE_KEY
7. Update your app to use axios or other http client of your choice to call functions at your fly.io URL instead of using the supabase client. You can get your JWT from the supabase client at client.auth.headers.Authorization and include it in your request to authenticate the same way you do on supabase edge functions.
8. Use bun --watch index.ts to run locally with appropriate env values to connect to local supabase such asSUPABASE_DB_URL=postgresql://postgres:postgres@localhost:54322/postgres

I've deployed this setup to production and so far it's vastly more stable and more performant. No more Deno pain either, I can import and use libraries and monorepo code normally and no more broken Deno language server. Hopefully the Supabase team will continue to make progress on edge functions and eventually they'll be more usable/stable (please no more Deno, it's an incredible pain) but this workaround was pretty easy and is working well so far.

[ouipark]

can you add an exemple please
?

[leblancmeneses]

After upgrading to the latest I started getting 546. My guess is it is related to supabase-edge-runtime upgrade.

Serving functions on http://127.0.0.1:54321/functions/v1/<function-name>
Using supabase-edge-runtime-1.36.1 (compatible with Deno v1.40.3)

{"code":"WORKER_LIMIT","message":"Worker failed to respond due to an error or resource limit (please check logs)"}%

I narrowed it down to

import { Hono } from 'hono/mod.ts';
import sql from '../core/db.ts';

const app = new Hono();

app.get('/', async (c) => {
  const result = await sql<any>`SELECT * FROM auth.users`;
  if (result.length === 0) {
    return c.json({ error: 'User not found' }, 404);
  }
  return c.text('Hello admin/index.ts!')
});

Once I add db.ts it begins to fail with the below error.

import postgres from "https://deno.land/x/[email protected]/mod.js"

const databaseUrl = Deno.env.get("SUPABASE_DB_URL")!;
const sql = postgres(databaseUrl, {
  max: 3,
  types: {
    bigint: postgres.BigInt
  }
});

export default sql;

DEBUG Specifier "https://deno.land/[email protected]/node/stream.ts" was not mapped in import map.
runtime has escaped from the event loop unexpectedly: event loop error: TypeError: internals.warnOnDeprecatedApi is not a function
    at get rid (ext:deno_io/12_io.js:277:15)
    at createWritableStdioStream (https://deno.land/[email protected]/node/_process/streams.mjs:29:21)
    at https://deno.land/[email protected]/node/_process/streams.mjs:68:38
failed to send request to user worker: connection error: Connection reset by peer (os error 104)
user worker failed to respond: connection error: Connection reset by peer (os error 104)
InvalidWorkerResponse: user worker failed to respond
    at async Promise.allSettled (index 1)
    at async UserWorker.fetch (ext:sb_user_workers/user_workers.js:79:21)
    at async Server.<anonymous> (file:///home/deno/main/index.ts:146:12)
    at async #respond (https://deno.land/[email protected]/http/server.ts:220:18) {
  name: "InvalidWorkerResponse"
}
DEBUG CPU time used: 217791ms

How can I opt out of supabase-edge-runtime-1.36.1 and go back to the original deno runtime or specific version like: edge-runtime:v1.33.5 ?

[nyannyacha]

Hello @leblancmeneses 😁

This issue is being tracked in PR-279 in the edge-runtime repository.
Thanks for reporting this issue!

[laktek]

@leblancmeneses Fix should be available in Supabase CLI versions v1.146.1 or above.

(Thanks @nyannyacha for the fix)

[pariah140]

I'm a bit confused because this version isn't available if I try to upgrade using brew. can you help?

[nyannyacha]

I'm a bit confused because this version isn't available if I try to upgrade using brew. can you help?

Hello @pariah140 😁

supabase tap seems does not track the version.
So if you want a specific version (currently v1.145.4 is the latest), you will need to install it via a package manager like npm.

[caseydierking]

@pariah140 , I just had this same issue. However, I was using NPM instead of homebrew. npm install [email protected] to install a specific version.

[derekchan916]

Is anyone still receiving this error? This is becoming more and more problematic for me and I can't seem to find a solution. I'm using "@supabase/supabase-js": "^2.22.0" and frequently see 520s and 546s, but my metrics are all under the limits.

The CPU time limit seems to have changed now in the docs, but still, I'm averaging 400ms CPU time.. https://supabase.com/docs/guides/functions/debugging#limitations
Memory limit for the function: 256MB
Wall clock limit: 400 s
CPU Time: 2 s

I wish the logs were better here. I just upgraded my supabase-js version to the latest and see if it's still happening.

On top of this, I am getting a lot of memory limit exceeded errors in the logs which I think are a different issue? But there are no logs for this aside, Memory limit exceeded, and then shut down.

[jonathanlal]

yeah having this issue also :'(