Issue upgrading from 1.172 to 1.173+

[jdanielsonivim]

I have a 1.172 container running with an external postresql database. I was able to update from 1.143 all the way to 1.172, but going from 1.172 to 1.173 I don't get beyond "connected to database" in logs. Anyone else run into this?

[rubenfiszel]

Hi, could you paste the entire logs of all your pods/services.

[jdanielsonivim]

There was almost no log, which I thought was weird. All I found was

[2m2023-12-16T21:15:24.144423Z�[0m �[32m INFO�[0m �[2mwindmill�[0m�[2m:�[0m Database connected

Then after that, it just hung. It is potentially user error, but I haven't ran into this problem before.

[rubenfiszel]

Is that the logs of all your containers or just one ?

[jdanielsonivim]

Here is more from the first of four I had launched

December 16, 2023 at 16:15 (UTC-5:00) | �[2m2023-12-16T21:15:18.174430Z�[0m �[32m INFO�[0m �[2msqlx::postgres::notice�[0m�[2m:�[0m relation "_sqlx_migrations" already exists, skipping | 5be14cb12c5746449ba0c41f86cb5fa4

December 16, 2023 at 16:15 (UTC-5:00) | �[2m2023-12-16T21:15:18.167455Z�[0m �[32m INFO�[0m �[2mwindmill�[0m�[2m:�[0m Database connected | 5be14cb12c5746449ba0c41f86cb5fa4

December 16, 2023 at 16:15 (UTC-5:00) | �[2m2023-12-16T21:15:18.146017Z�[0m �[32m INFO�[0m �[2mwindmill�[0m�[2m:�[0m Connecting to database... | 5be14cb12c5746449ba0c41f86cb5fa4

[jdanielsonivim]

here is dump of sqlx table
data-1702779242393.csv

[rubenfiszel]

My suspicion is that one of your container that is not among the one you're showing is doing a migration that is slow and the other containers are waiting on that to continue further Le dim. 17 déc. 2023, 11:15, jdanielsonivim ***@***.***> a écrit :

…

here is dump of sqlx table data-1702779242393.csv <https://github.com/windmill-labs/windmill/files/13694479/data-1702779242393.csv> — Reply to this email directly, view it on GitHub <#2870 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACDJAGKLPQ4JQPHGUJSPTTYJZITPAVCNFSM6AAAAABAX3IDM2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TQNZUGI3DK> . You are receiving this because you commented.Message ID: ***@***.***>

[jdanielsonivim]

Okay. Thanks. I'm spinning up a replicated db server of the 1.172 and I'm going to launch a single 1.173 server to try and narrow in

[jdanielsonivim]

Is there an estimated duration for the migration? Not a ton of data in the db. But your suggestion made me think that my healthcheck maybe cycling it out before the upgrade finishes

[jdanielsonivim]

After looking at the source for that version, I've narrowed the problem down to 20230914082838_relax_completed_job_constraints.up.sql just hanging when it runs.

ALTER TABLE completed_job ALTER COLUMN duration_ms TYPE bigint;
ALTER TABLE completed_job ALTER COLUMN email TYPE VARCHAR(255);
ALTER TABLE queue ALTER COLUMN email TYPE VARCHAR(255);
ALTER TABLE queue ALTER COLUMN canceled_by TYPE VARCHAR(255);

[rubenfiszel]

It's not hanging, it's just executing the migration (which may take some time)

Having less jobs in completed_job table or a faster db might help

[jdanielsonivim]

yeah, that's it. That table was much larger than anticipated. I had thought it had a limit of days to keep past entries, but I can't find where that was set, or am I'm misremembering?

[rubenfiszel]

Once you have upgraded, go to your instance settings, you will see configurable in the core as "job retention" Le dim. 17 déc. 2023, 13:22, jdanielsonivim ***@***.***> a écrit :

…

yeah, that's it. That table was much larger than anticipated. I had thought it had a limit of days to keep past entries, but I can't find where that was set, or am I'm misremembering? — Reply to this email directly, view it on GitHub <#2870 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACDJAGNUMUVLRQ7AJPBQ6TYJZXQXAVCNFSM6AAAAABAX3IDM2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TQNZUGYZTK> . You are receiving this because you commented.Message ID: ***@***.***>