Issue tracking for flyte binary experience

[wild-endeavor]

Personal notes when using flyte binary. Not enough to make a GH issue for it yet.

DB collision on start

There seems to be a race condition when starting up the single executable - datacatalog may try to create the same postgres db and fail. Should amend datacatalog code. Should catch and ignore this error.

$ kf logs flytebinary-5b856d5cf5-rz5s5 -p
time="2022-10-07T20:50:47Z" level=info msg="Using config file: [/etc/flyte/flyte.yaml]"
{"json":{"src":"start.go:62"},"level":"info","msg":"Running Database Migrations...","ts":"2022-10-07T20:50:47Z"}

2022/10/07 20:50:47 /go/pkg/mod/github.com/flyteorg/[email protected]/pkg/repositories/database.go:113
[error] failed to initialize database, got error failed to connect to `host=127.0.0.1 user=postgres database=flyteadmin`: server error (FATAL: database "flyteadmin" does not exist (SQLSTATE 3D000))
{"json":{"src":"database.go:122"},"level":"warning","msg":"Database [flyteadmin] does not exist","ts":"2022-10-07T20:50:47Z"}

2022/10/07 20:50:47 /go/pkg/mod/github.com/flyteorg/[email protected]/pkg/repositories/config/postgres.go:73
[error] failed to initialize database, got error failed to connect to `host=127.0.0.1 user=postgres database=flyteadmin`: server error (FATAL: database "flyteadmin" does not exist (SQLSTATE 3D000))
{"json":{"src":"initialize.go:39"},"level":"warning","msg":"Database [flyteadmin] does not exist, trying to create it now","ts":"2022-10-07T20:50:47Z"}
{"json":{"src":"database.go:135"},"level":"info","msg":"Creating database flyteadmin","ts":"2022-10-07T20:50:47Z"}
{"json":{"src":"handle.go:59"},"level":"info","msg":"Creating Database flyteadmin since it does not exist","ts":"2022-10-07T20:50:47Z"}
{"json":{"src":"initialize.go:52"},"level":"error","msg":"Failed to create DB flyteadmin err ERROR: duplicate key value violates unique constraint \"pg_database_datname_index\" (SQLSTATE 23505)","ts":"2022-10-07T20:50:47Z"}

2022/10/07 20:50:47 /go/pkg/mod/github.com/flyteorg/[email protected]/pkg/repositories/handle.go:63 ERROR: duplicate key value violates unique constraint "pg_database_datname_index" (SQLSTATE 23505)
[91.925ms] [rows:0] CREATE DATABASE flyteadmin
panic: ERROR: duplicate key value violates unique constraint "pg_database_datname_index" (SQLSTATE 23505)

goroutine 13 [running]:
github.com/flyteorg/datacatalog/pkg/repositories.Migrate({0x54e0280, 0xc000137840})
        /go/pkg/mod/github.com/flyteorg/[email protected]/pkg/repositories/initialize.go:53 +0x62e
github.com/flyteorg/flyte/cmd/single.startDataCatalog({0x54e0280, 0xc000137840}, {0x0?})
        /flyteorg/build/cmd/single/start.go:42 +0x27
github.com/flyteorg/flyte/cmd/single.glob..func4.3()
        /flyteorg/build/cmd/single/start.go:178 +0x37
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x67
created by golang.org/x/sync/errgroup.(*Group).Go
        /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x8