Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: use _supabase database for internals #2707

Merged
merged 11 commits into from
Sep 27, 2024
Merged

chore: use _supabase database for internals #2707

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
bb28463
chore: use _supabase database for internals
avallete Sep 24, 2024
5e94687
chore: fix tests mocks
avallete Sep 24, 2024
5904062
Merge branch 'develop' into avallete/chore-create-new-supabase-manage…
avallete Sep 24, 2024
e6788f3
chore: add realtime schema as well
avallete Sep 24, 2024
76ef3d2
fix: make the start work for postgres 14 and 13
avallete Sep 24, 2024
edb664a
Merge branch 'develop' into avallete/chore-create-new-supabase-manage…
avallete Sep 25, 2024
49fd57f
Merge branch 'develop' into avallete/chore-create-new-supabase-manage…
avallete Sep 25, 2024
2822808
Merge branch 'develop' into avallete/chore-create-new-supabase-manage…
avallete Sep 26, 2024
ccf214d
chore: restore realtime to postgres database
avallete Sep 27, 2024
e911718
chore: remove relatime migration
avallete Sep 27, 2024
3115143
Update internal/db/start/start.go
avallete Sep 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions internal/db/dump/dump.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ func dumpData(ctx context.Context, config pgconn.Config, schema, excludeTable []
// "storage",
// "supabase_functions",
"supabase_migrations",
// TODO: Remove in a few version in favor of _supabase internal db
"_analytics",
"_realtime",
"_supavisor",
Expand Down
2 changes: 2 additions & 0 deletions internal/db/reset/reset.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,8 @@ func recreateDatabase(ctx context.Context, options ...func(*pgx.ConnConfig)) err
Statements: []string{
"DROP DATABASE IF EXISTS postgres WITH (FORCE)",
"CREATE DATABASE postgres WITH OWNER postgres",
"DROP DATABASE IF EXISTS _supabase WITH (FORCE)",
"CREATE DATABASE _supabase WITH OWNER postgres",
},
}
return sql.ExecBatch(ctx, conn)
Expand Down
11 changes: 9 additions & 2 deletions internal/db/reset/reset_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,10 @@ func TestRecreateDatabase(t *testing.T) {
Query("DROP DATABASE IF EXISTS postgres WITH (FORCE)").
Reply("DROP DATABASE").
Query("CREATE DATABASE postgres WITH OWNER postgres").
Reply("CREATE DATABASE").
Query("DROP DATABASE IF EXISTS _supabase WITH (FORCE)").
Reply("DROP DATABASE").
Query("CREATE DATABASE _supabase WITH OWNER postgres").
Reply("CREATE DATABASE")
// Run test
assert.NoError(t, recreateDatabase(context.Background(), conn.Intercept))
Expand Down Expand Up @@ -194,8 +198,11 @@ func TestRecreateDatabase(t *testing.T) {
Reply("DO").
Query("DROP DATABASE IF EXISTS postgres WITH (FORCE)").
ReplyError(pgerrcode.ObjectInUse, `database "postgres" is used by an active logical replication slot`).
Query("CREATE DATABASE postgres WITH OWNER postgres")
// Run test
Query("CREATE DATABASE postgres WITH OWNER postgres").
Query("DROP DATABASE IF EXISTS _supabase WITH (FORCE)").
Reply("DROP DATABASE").
Query("CREATE DATABASE _supabase WITH OWNER postgres").
Reply("CREATE DATABASE")
err := recreateDatabase(context.Background(), conn.Intercept)
// Check error
assert.ErrorContains(t, err, `ERROR: database "postgres" is used by an active logical replication slot (SQLSTATE 55006)`)
Expand Down
13 changes: 10 additions & 3 deletions internal/db/start/start.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ var (
HealthTimeout = 120 * time.Second
//go:embed templates/schema.sql
initialSchema string
//go:embed templates/_supabase.sql
_supabaseSchema string
)

func Run(ctx context.Context, fsys afero.Fs) error {
Expand Down Expand Up @@ -82,7 +84,7 @@ func NewContainerConfig() container.Config {
Retries: 3,
},
Entrypoint: []string{"sh", "-c", `cat <<'EOF' > /etc/postgresql.schema.sql && cat <<'EOF' > /etc/postgresql-custom/pgsodium_root.key && docker-entrypoint.sh postgres -D /etc/postgresql
` + initialSchema + `
` + initialSchema + "\n" + _supabaseSchema + "\n" + `
avallete marked this conversation as resolved.
Show resolved Hide resolved
EOF
` + utils.Config.Db.RootKey + `
EOF
Expand Down Expand Up @@ -122,7 +124,12 @@ func StartDatabase(ctx context.Context, fsys afero.Fs, w io.Writer, options ...f
},
}
if utils.Config.Db.MajorVersion <= 14 {
config.Entrypoint = nil
config.Entrypoint = []string{"sh", "-c", `
cat <<'EOF' > /docker-entrypoint-initdb.d/supabase_schema.sql
` + _supabaseSchema + `
EOF
docker-entrypoint.sh postgres -D /etc/postgresql
`}
avallete marked this conversation as resolved.
Show resolved Hide resolved
hostConfig.Tmpfs = map[string]string{"/docker-entrypoint-initdb.d": ""}
}
// Creating volume will not override existing volume, so we must inspect explicitly
Expand Down Expand Up @@ -228,7 +235,7 @@ func initRealtimeJob(host string) utils.DockerJob {
"DB_PORT=5432",
"DB_USER=supabase_admin",
"DB_PASSWORD=" + utils.Config.Db.Password,
"DB_NAME=postgres",
"DB_NAME=_supabase",
"DB_AFTER_CONNECT_QUERY=SET search_path TO _realtime",
"DB_ENC_KEY=" + utils.Config.Realtime.EncryptionKey,
"API_JWT_SECRET=" + utils.Config.Auth.JwtSecret,
Expand Down
18 changes: 18 additions & 0 deletions internal/db/start/templates/_supabase.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
CREATE DATABASE _supabase WITH OWNER postgres;

-- Switch to the newly created _supabase database
\c _supabase
-- Create schemas in _supabase database for
-- internals tools and reports to not overload user database
-- with non-user activity
CREATE SCHEMA IF NOT EXISTS _realtime;
ALTER SCHEMA _realtime OWNER TO postgres;
CREATE SCHEMA IF NOT EXISTS realtime;
ALTER SCHEMA realtime OWNER TO postgres;
avallete marked this conversation as resolved.
Show resolved Hide resolved

CREATE SCHEMA IF NOT EXISTS _analytics;
ALTER SCHEMA _analytics OWNER TO postgres;

CREATE SCHEMA IF NOT EXISTS _supavisor;
ALTER SCHEMA _supavisor OWNER TO postgres;
\c postgres
9 changes: 0 additions & 9 deletions internal/db/start/templates/schema.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,6 @@ ALTER USER supabase_storage_admin WITH PASSWORD :'pgpass';
ALTER USER supabase_replication_admin WITH PASSWORD :'pgpass';
ALTER USER supabase_read_only_user WITH PASSWORD :'pgpass';

create schema if not exists _realtime;
alter schema _realtime owner to postgres;

create schema if not exists _analytics;
alter schema _analytics owner to postgres;

create schema if not exists _supavisor;
alter schema _supavisor owner to postgres;

BEGIN;

-- Create pg_net extension
Expand Down
8 changes: 4 additions & 4 deletions internal/start/start.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ func run(p utils.Program, ctx context.Context, fsys afero.Fs, excludedContainers
// Start Logflare
if utils.Config.Analytics.Enabled && !isContainerExcluded(utils.Config.Analytics.Image, excluded) {
env := []string{
"DB_DATABASE=" + dbConfig.Database,
"DB_DATABASE=_supabase",
"DB_HOSTNAME=" + dbConfig.Host,
fmt.Sprintf("DB_PORT=%d", dbConfig.Port),
"DB_SCHEMA=_analytics",
Expand Down Expand Up @@ -228,7 +228,7 @@ func run(p utils.Program, ctx context.Context, fsys afero.Fs, excludedContainers
)
case config.LogflarePostgres:
env = append(env,
fmt.Sprintf("POSTGRES_BACKEND_URL=postgresql://%s:%s@%s:%d/%s", dbConfig.User, dbConfig.Password, dbConfig.Host, dbConfig.Port, dbConfig.Database),
fmt.Sprintf("POSTGRES_BACKEND_URL=postgresql://%s:%s@%s:%d/%s", dbConfig.User, dbConfig.Password, dbConfig.Host, dbConfig.Port, "_supabase"),
"POSTGRES_BACKEND_SCHEMA=_analytics",
)
}
Expand Down Expand Up @@ -750,7 +750,7 @@ EOF
fmt.Sprintf("DB_PORT=%d", dbConfig.Port),
"DB_USER=supabase_admin",
"DB_PASSWORD=" + dbConfig.Password,
"DB_NAME=" + dbConfig.Database,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

realtime schema should be initialised inside the postgres database. We probably need to check with @filipecabaco if there's additional config for separating the control plane database.

psql (16.3, server 15.6)
Type "help" for help.

_supabase=> \dn
        List of schemas
    Name    |       Owner
------------+-------------------
 _analytics | postgres
 _realtime  | postgres
 _supavisor | postgres
 public     | pg_database_owner
 realtime   | postgres
(5 rows)

_supabase=> \dt realtime.*
                       List of relations
  Schema  |       Name        | Type  |          Owner
----------+-------------------+-------+-------------------------
 realtime | messages          | table | supabase_realtime_admin
 realtime | schema_migrations | table | supabase_admin
 realtime | subscription      | table | supabase_admin
(3 rows)

"DB_NAME=_supabase",
"DB_AFTER_CONNECT_QUERY=SET search_path TO _realtime",
"DB_ENC_KEY=" + utils.Config.Realtime.EncryptionKey,
"API_JWT_SECRET=" + utils.Config.Auth.JwtSecret,
Expand Down Expand Up @@ -1045,7 +1045,7 @@ EOF
"PORT=4000",
fmt.Sprintf("PROXY_PORT_SESSION=%d", portSession),
fmt.Sprintf("PROXY_PORT_TRANSACTION=%d", portTransaction),
fmt.Sprintf("DATABASE_URL=ecto://%s:%s@%s:%d/%s", dbConfig.User, dbConfig.Password, dbConfig.Host, dbConfig.Port, dbConfig.Database),
fmt.Sprintf("DATABASE_URL=ecto://%s:%s@%s:%d/%s", dbConfig.User, dbConfig.Password, dbConfig.Host, dbConfig.Port, "_supabase"),
"CLUSTER_POSTGRES=true",
"SECRET_KEY_BASE=" + utils.Config.Db.Pooler.SecretKeyBase,
"VAULT_ENC_KEY=" + utils.Config.Db.Pooler.EncryptionKey,
Expand Down