Removal of app.settings.jwt_secret from the database

[staaldraad]

Introduction

We are removing app.settings.jwt_secret from the postgres database on 2024/11/22.

This setting has previously been available through our PostgREST integration, and could be accessed using current_setting('app.settings.jwt_secret') in SQL.

Why are we doing this?

The jwt_secret can be used to mint new, custom JWTs and is security sensitive. Supabase limits access to the jwt_secret , through both the dashboard and API, to specific roles (owner, admin and developer). Allowing access to this setting directly in the database can allow bypassing of these restrictions.

What do you need to do?

If you need the jwt_secret, it can be retrieved through the Supabase dashboard.

If you are using the app.settings.jwt_secret in SQL, you will need to update your function to retrieve this value from Vault.

select vault.create_secret('JWT_SECRET', 'app.jwt_secret', 'The jwt secret');

-- retrieve the value, this replaces select current_setting('app.settings.jwt_secret')
select decrypted_secret 
   from vault.decrypted_secrets 
   where name = 'app.jwt_secret';

Also, please consult the changelog entry for Asymmetric Keys to understand the coming changes to jwt_secret and how keys at Supabase are changing.

[rkistner]

@staaldraad While it seems like right now we can still access the secret, the announced date (2024/11/22) is zero notice.

I'm with PowerSync, and we currently rely on app.settings.jwt_secret for integrating with Supabase auth. While I think the new asymmetric keys functionality is a much better way to do this, it is not even available yet.

Is it possible to delay this change until a asymmetric keys have been rolled out, and users had a chance to switch over? As is, this change will break a couple hundred PowerSync projects integrated with Supabase.

[staaldraad]

Hey @rkistner

We understand this is short notice, with security related changes our timelines are shortened. However, this decision was not made blindly and was based on usage data. We reached out to projects where we detected usage of this parameter, it is unfortunate that we did not manage to catch PowerSync in this communication.

The change has been applied to all new projects, and we will soon roll this out to existing projects.

To mimic and maintain the existing mechanism of using current_setting(), the following can be used (although we recommend Vault usage or a table only the postgres user has access to):

As the postgres user:

alter role postgres in database postgres set pgrst.app.jwt_secret to "JWT_FROM_DASHBOARD";

Any new session can then pickup the value via:

select current_setting('pgrst.app.jwt_secret');

This value will not be synced or updated by Supabase. Fortunately, JWT Secret resetting is not a frequent operation.

The changes for Asymmetric JWTs should start becoming public soon and this will include exposing a JWKS end-point, which I believe will allow PowerSync to work without any changes.

[carlos-alberto]

@staaldraad This has broken our production workflows. I cannot find any notification that this was happening. This is very disappointing from Supabase.

Also, the vault which is suggested as a replacement is currently in alpha. This does not inspire confidence for critical secrets.

[w3b6x9]

This has broken our production workflows. I cannot find any notification that this was happening. This is very disappointing from Supabase.

we're very sorry that this broke your production system. as it involved removing jwt secret from your database, we wanted to send an email notification to only those customers that had accessed the jwt secret via database queries. unfortunately, the logs we referred to when generating the list of projects to contact were incomplete so we ended up emailing a subset of project owners.

we have since changed our internal practices and in the future we will be emailing all project owners, emailing multiple times over time in case any owners missed any, and give ample time for project owners to introduce a workaround.

[jshearer]

This just caused a production incident for us that we had to scramble to hotfix @staaldraad @KevinBrolly @awalias. The alter role command you posted does not fix the problem as we were using show app.settings.jwt_secret and not current_setting().

Short turn-around for security-related changes is fine, but the lack of any heads-up caused us some serious pain here. We had to find this discussion by googling unrecognized configuration parameter "app.settings.jwt_secret".

[dyaffe]

How was this announced via a github discussion without further communciation? It's currently causing major downtime for us.

[staaldraad]

hey @jshearer @dyaffe

Sorry to read this had unintended consequences.

The alter role command will not be "seen" by open connections as role-specific variable settings only take effect at login. Unfortunately, these connections will need to be re-established.

As to the communication of this, in the weeks leading up to the change we did reach out directly to customers who we detected using this parameter. We are looking into why this communication may not have been sent for your projects.

[carlos-alberto]

For something with such potential impact why not send a communication to all projects?

[dyaffe]

@staaldraad we had to take care of this on our own and figured out a solution but it cause major production impact -- our largest downtime ever and some very unhappy customers.

We were given no advance notice of it which is particularly troubling and need to consider self hosting Supabase now because we can't risk a future incident of the sort.

[w3b6x9]

This has broken our production workflows. I cannot find any notification that this was happening. This is very disappointing from Supabase.

we're very sorry that this broke your production system. as it involved removing jwt secret from your database, we wanted to send an email notification to only those customers that had accessed the jwt secret via database queries. unfortunately, the logs we referred to when generating the list of projects to contact were incomplete so we ended up emailing a subset of project owners.

we have since changed our internal practices and in the future we will be emailing all project owners, emailing multiple times over time in case any owners missed any, and give ample time for project owners to introduce a workaround.

• https://github.com/orgs/supabase/discussions/30606#discussioncomment-11635101

[PhilippS93]

This also broke our paid production setting!
I double checked emails and nhotifications, there was no notice.

We were migrating now to the vault setting.

[w3b6x9]

This has broken our production workflows. I cannot find any notification that this was happening. This is very disappointing from Supabase.

we're very sorry that this broke your production system. as it involved removing jwt secret from your database, we wanted to send an email notification to only those customers that had accessed the jwt secret via database queries. unfortunately, the logs we referred to when generating the list of projects to contact were incomplete so we ended up emailing a subset of project owners.

we have since changed our internal practices and in the future we will be emailing all project owners, emailing multiple times over time in case any owners missed any, and give ample time for project owners to introduce a workaround.

• https://github.com/orgs/supabase/discussions/30606#discussioncomment-11635101