Plugin Server - Node Error

[Shauryadhaka]

Bug Description

Stuck at the validation screen post a fresh install via the curl command for self hosted posthog. This seemed to work when i did the same steps around 1.5 weeks back.

Please describe.
If this affects the front-end, screenshots would be of great help.

If you are on PostHog Cloud it would be really valuable if you can share any links where the problem occurs. This speeds up our ability to troubleshoot tremendously.

How to reproduce

1. Install posthog on ubuntu with docker via this command
   /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/posthog/posthog/HEAD/bin/deploy-hobby)"
2. Setup is complete and when i open the posthog url it shows me a screen something like this
   
3. plugin node container logs shows nothing as breaking
   {"level":"info","time":1741602814687,"pid":49,"hostname":"4408dc834deb","batchSize":0,"msg":"[MAIN] 🔁 session_batch_manager_flushing"} {"level":"info","time":1741602814687,"pid":49,"hostname":"4408dc834deb","partitions":0,"totalSize":0,"msg":"[MAIN] 🔁 session_batch_recorder_flushing"} {"level":"info","time":1741602814687,"pid":49,"hostname":"4408dc834deb","msg":"[MAIN] 🔁 session_batch_recorder_flushed_no_sessions"} {"level":"info","time":1741602819357,"pid":49,"hostname":"4408dc834deb","groupId":"clickhouse-ingestion","topic":"events_plugin_ingestion","messagesPerSecond":0,"batchesProcessed":0,"lastHeartbeatTime":"2025-03-10T10:33:38.400Z","msg":"[MAIN] 🔁 main_loop"}
4. DB logs looks like failed migrations thou
   {"level":"error","time":1741600857113,"pid":23,"hostname":"4408dc834deb","error":"error: relation \"posthog_pluginattachment\" does not exist\n at Parser.parseErrorMessage (/code/node_modules/.pnpm/[email protected]/node_modules/pg-protocol/dist/parser.js:287:98)\n at Parser.handlePacket (/code/node_modules/.pnpm/[email protected]/node_modules/pg-protocol/dist/parser.js:126:29)\n at Parser.parse (/code/node_modules/.pnpm/[email protected]/node_modules/pg-protocol/dist/parser.js:39:38)\n at Socket.<anonymous> (/code/node_modules/.pnpm/[email protected]/node_modules/pg-protocol/dist/index.js:11:42)\n at Socket.emit (node:events:517:28)\n at Socket.emit (node:domain:489:12)\n at addChunk (node:internal/streams/readable:368:12)\n at readableAddChunk (node:internal/streams/readable:341:9)\n at Readable.push (node:internal/streams/readable:278:10)\n at TCP.onStreamRead (node:internal/stream_base_commons:190:23)","msg":"[MAIN] :boom: Launchpad failure!"}

Not sure what is wrong here

Additional context

This was a fresh installation via the steps mentioned in the docs
Output of error docker logs grep

2025-03-11T10:38:47.765677Z [warning ] Skipping async migrations setup. This is unsafe in production! [posthog.apps] pid=10 tid=136746441149312 2025-03-11T10:38:47.787755Z [info ] AXES: BEGIN LOG [axes.apps] pid=10 tid=136746441149312 2025-03-11T10:38:47.790881Z [info ] AXES: Using django-axes version 5.9.0 [axes.apps] pid=10 tid=136746441149312 2025-03-11T10:38:47.795192Z [info ] AXES: blocking by IP only. [axes.apps] pid=10 tid=136746441149312 2025-03-11T10:39:40.019219Z [warning ] Skipping async migrations setup. This is unsafe in production! [posthog.apps] pid=23 tid=132635376798592 2025-03-11T10:39:40.035075Z [info ] AXES: BEGIN LOG [axes.apps] pid=23 tid=132635376798592 2025-03-11T10:39:40.038303Z [info ] AXES: Using django-axes version 5.9.0 [axes.apps] pid=23 tid=132635376798592 2025-03-11T10:39:40.038582Z [info ] AXES: blocking by IP only. [axes.apps] pid=23 tid=132635376798592 2025-03-11T10:40:24.735262Z [warning ] Skipping async migrations setup. This is unsafe in production! [posthog.apps] pid=36 tid=125392359848832 2025-03-11T10:40:24.752198Z [info ] AXES: BEGIN LOG [axes.apps] pid=36 tid=125392359848832 2025-03-11T10:40:24.756682Z [info ] AXES: Using django-axes version 5.9.0 [axes.apps] pid=36 tid=125392359848832 2025-03-11T10:40:24.760509Z [info ] AXES: blocking by IP only. [axes.apps] pid=36 tid=125392359848832 (node:49) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. Please migrate your code to use AWS SDK for JavaScript (v3). For more information, check the migration guide at https://a.co/7PzMCcy (Use node --trace-warnings ... to show where the warning was created) {"level":"info","time":1741689642462,"pid":49,"hostname":"4acb68f902c9","error":null,"brokers":[{"id":1001,"host":"kafka","port":9092}],"msg":"[MAIN] 📝 librdkafka producer connected"} {"level":"error","time":1741689642566,"pid":49,"hostname":"4acb68f902c9","msg":"[MAIN] 💥 Cyclotron database URL not set."} {"level":"error","time":1741689645785,"pid":49,"hostname":"4acb68f902c9","msg":"[MAIN] 💥 Cyclotron database URL not set."} {"level":"info","time":1741689646015,"pid":49,"hostname":"4acb68f902c9","error":null,"brokers":[{"id":1001,"host":"kafka","port":9092}],"msg":"[MAIN] 📝 librdkafka producer connected"} {"level":"info","time":1741689646016,"pid":49,"hostname":"4acb68f902c9","error":null,"brokers":[{"id":1001,"host":"kafka","port":9092}],"msg":"[MAIN] 📝 librdkafka producer connected"} {"level":"error","time":1741689646366,"pid":49,"hostname":"4acb68f902c9","bucket":"posthog","error":{"errno":-111,"code":"ECONNREFUSED","syscall":"connect","address":"127.0.0.1","port":19000,"$metadata":{"attempts":3,"totalRetryDelay":87}},"msg":"[MAIN] 🔁 s3_session_batch_writer_healthcheck_error"} {"level":"info","time":1741689646379,"pid":49,"hostname":"4acb68f902c9","error":null,"brokers":[{"id":1001,"host":"kafka","port":9092}],"msg":"[MAIN] 📝 librdkafka producer connected"} {"level":"info","time":1741689646387,"pid":49,"hostname":"4acb68f902c9","error":null,"brokers":[{"id":1001,"host":"kafka","port":9092}],"msg":"[MAIN] 📝 librdkafka producer connected"} {"level":"info","time":1741689646390,"pid":49,"hostname":"4acb68f902c9","error":null,"brokers":[{"id":1001,"host":"kafka","port":9092}],"msg":"[MAIN] 📝 librdkafka producer connected"} {"level":"info","time":1741689646689,"pid":49,"hostname":"4acb68f902c9","error":null,"brokers":[{"id":1001,"host":"kafka","port":9092}],"msg":"[MAIN] 📝 librdkafka producer connected"} {"level":"info","time":1741689646690,"pid":49,"hostname":"4acb68f902c9","error":null,"brokers":[{"id":1001,"host":"kafka","port":9092}],"msg":"[MAIN] 📝 librdkafka producer connected"} {"level":"info","time":1741689646693,"pid":49,"hostname":"4acb68f902c9","error":null,"brokers":[{"id":1001,"host":"kafka","port":9092}],"msg":"[MAIN] 📝 librdkafka producer connected"} {"level":"info","time":1741689646699,"pid":49,"hostname":"4acb68f902c9","error":null,"brokers":[{"id":1001,"host":"kafka","port":9092}],"msg":"[MAIN] 📝 librdkafka producer connected"} {"level":"info","time":1741689646713,"pid":49,"hostname":"4acb68f902c9","error":null,"brokers":[{"id":1001,"host":"kafka","port":9092}],"msg":"[MAIN] 📝 librdkafka producer connected"} {"level":"info","time":1741689646718,"pid":49,"hostname":"4acb68f902c9","error":null,"brokers":[{"id":1001,"host":"kafka","port":9092}],"msg":"[MAIN] 📝 librdkafka producer connected"} {"level":"info","time":1741689646722,"pid":49,"hostname":"4acb68f902c9","error":null,"brokers":[{"id":1001,"host":"kafka","port":9092}],"msg":"[MAIN] 📝 librdkafka producer connected"} {"level":"info","time":1741689646722,"pid":49,"hostname":"4acb68f902c9","error":null,"brokers":[{"id":1001,"host":"kafka","port":9092}],"msg":"[MAIN] 📝 librdkafka producer connected"} {"level":"info","time":1741689646726,"pid":49,"hostname":"4acb68f902c9","error":null,"brokers":[{"id":1001,"host":"kafka","port":9092}],"msg":"[MAIN] 📝 librdkafka producer connected"} {"level":"info","time":1741689646743,"pid":49,"hostname":"4acb68f902c9","error":null,"brokers":[{"id":1001,"host":"kafka","port":9092}],"msg":"[MAIN] 📝 librdkafka producer connected"} {"level":"info","time":1741689648052,"pid":49,"hostname":"4acb68f902c9","err":{"type":"LibrdKafkaError","message":"Local: Assign partitions","stack":"Error: Local: Assign partitions\n at Function.createLibrdkafkaError [as create] (/code/node_modules/.pnpm/[email protected]_patch_hash=bugorwxdhlhl2utknko2c5ibqm/node_modules/node-rdkafka/lib/error.js:456:10)\n at KafkaConsumer.cooperativeRebalanceCallback (/code/node_modules/.pnpm/[email protected]_patch_hash=bugorwxdhlhl2utknko2c5ibqm/node_modules/node-rdkafka/lib/kafka-consumer.js:45:25)","code":-175,"errno":-175,"origin":"kafka"},"topicPartitions":[{"topic":"session_recording_snapshot_item_events","partition":0}],"msg":"[MAIN] 🔁 blob_ingester_consumer - rebalancing"} {"level":"info","time":1741689648086,"pid":49,"hostname":"4acb68f902c9","err":{"type":"LibrdKafkaError","message":"Local: Assign partitions","stack":"Error: Local: Assign partitions\n at Function.createLibrdkafkaError [as create] (/code/node_modules/.pnpm/[email protected]_patch_hash=bugorwxdhlhl2utknko2c5ibqm/node_modules/node-rdkafka/lib/error.js:456:10)\n at KafkaConsumer.cooperativeRebalanceCallback (/code/node_modules/.pnpm/[email protected]_patch_hash=bugorwxdhlhl2utknko2c5ibqm/node_modules/node-rdkafka/lib/kafka-consumer.js:45:25)","code":-175,"errno":-175,"origin":"kafka"},"topicPartitions":[{"topic":"session_recording_snapshot_item_events","partition":0}],"msg":"[MAIN] 🔁 blob_ingester_consumer_v2 - rebalancing"}

Debug info

- [ ] PostHog Cloud, Debug information: [please copy/paste from https://us.posthog.com/settings/project-details#variables or https://eu.posthog.com/settings/project-details#variables]
- [ ] PostHog Hobby self-hosted with `docker compose`, version/commit: latest version
- [ ] PostHog self-hosted with Kubernetes (deprecated, see [`Sunsetting Kubernetes support`](https://posthog.com/blog/sunsetting-helm-support-posthog)), version/commit: [please provide]

[Shauryadhaka]

Also if i upgrade it using this command
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/posthog/posthog/HEAD/bin/upgrade-hobby)"

The service starts giving 502 and caddy logs shows the same.

[sivagao]

same

[Shauryadhaka]

@pauldambra hey can you help us here. Saw an old thread which you solved earlier around the same thing

[freeResumeOnline]

Has anyone found a solution for this? I'm facing the same issue.

[Shauryadhaka]

docker-compose logs is also showing this
worker-1 | 2025-03-12T07:04:24.901856Z [error ] task_failed [django_structlog.celery.receivers] error=relation "graphile_worker.jobs" does not exist worker-1 | psycopg.errors.UndefinedTable: relation "graphile_worker.jobs" does not exist worker-1 | with self.db.wrap_database_errors:

[Shauryadhaka]

Instance settings looks like this