All sources stuck on "Task will run immediately"

[ArGJolan]

I'm using tubesync version 0.13.6 with postgres as a database and I have around 30 video sources.

Every now and then, the Index media from source tasks get stuck with the status Task will run immediately.

After checking the logs, the only worrying thing I see is the following:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.OperationalError: SSL connection has been closed unexpectedly


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/background_task/tasks.py", line 43, in bg_runner
    func(*args, **kwargs)
  File "/app/sync/tasks.py", line 177, in index_source_task
    source.save()
  File "/usr/local/lib/python3.11/dist-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/usr/local/lib/python3.11/dist-packages/django/db/models/base.py", line 787, in save_base
    post_save.send(
  File "/usr/local/lib/python3.11/dist-packages/django/dispatch/dispatcher.py", line 180, in send
    return [
           ^
  File "/usr/local/lib/python3.11/dist-packages/django/dispatch/dispatcher.py", line 181, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/sync/signals.py", line 72, in source_post_save
    media.save()
  File "/usr/local/lib/python3.11/dist-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/usr/local/lib/python3.11/dist-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
              ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/django/db/models/base.py", line 858, in _save_table
    updated = self._do_update(base_qs, using, pk_val, values, update_fields,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/django/db/models/base.py", line 912, in _do_update
    return filtered._update(values) > 0
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/django/db/models/query.py", line 802, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/django/db/models/sql/compiler.py", line 1559, in execute_sql
    cursor = super().execute_sql(result_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.11/dist-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/django/db/backends/utils.py", line 79, in _execute
    with self.db.wrap_database_errors:
  File "/usr/local/lib/python3.11/dist-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.11/dist-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.OperationalError: SSL connection has been closed unexpectedly

Failed to retrieve tasks. Database unreachable.
Failed to retrieve tasks. Database unreachable.
Failed to retrieve tasks. Database unreachable.
Failed to retrieve tasks. Database unreachable.
Failed to retrieve tasks. Database unreachable.
Failed to retrieve tasks. Database unreachable.
Failed to retrieve tasks. Database unreachable.
Failed to retrieve tasks. Database unreachable.
Failed to retrieve tasks. Database unreachable.
Failed to retrieve tasks. Database unreachable.
Failed to retrieve tasks. Database unreachable.
Failed to retrieve tasks. Database unreachable.
Failed to retrieve tasks. Database unreachable.

Which I assume happens when my postgres VM shuts down during backup (around 1 minute of downtime per day).
I wouldn't expect tubesync to do anything when database is unreachable but when it's up again it seems to be unable to recover automatically.

I tried restarting tubesync with no success.

Until now I fixed the issue multiple times by resetting tasks but it's not sustainable.

Thank you for your time developing / maintaining this project & for your help.

[meeb]

Yes that error suggests tubesync can't connect to your Postgres database server. There's no other really useful information in your logs other than basically "connection to Postgres failed" so I can't help further really. The workers may not reconnect properly to Postgres if the Postgres server itself restarts so that's something I can probably check.