[server] how do i delete a msg from the queue ?

[michabbb]

background: i have a failed msg for a specific event type.
i deleted the event type. now there is a msg stuck in the system I cannot deliver (I guess this is the reason) and the server is showing:

2024-09-28T19:40:38.291838Z ERROR process_queue_task{task_type="MessageV1"}: svix_server::worker: Unexpected: message doesn't exist msg_2mhUtDVDnpXDnEGUpggldlAKTT9
2024-09-28T19:41:23.772251Z ERROR process_queue_task{task_type="MessageV1"}: svix_server::worker: Unexpected: message doesn't exist msg_2mhUtDVDnpXDnEGUpggldlAKTT9
2024-09-28T19:42:09.050484Z ERROR process_queue_task{task_type="MessageV1"}: svix_server::worker: Unexpected: message doesn't exist msg_2mhUtDVDnpXDnEGUpggldlAKTT9
2024-09-28T19:42:54.322623Z ERROR process_queue_task{task_type="MessageV1"}: svix_server::worker: Unexpected: message doesn't exist msg_2mhUtDVDnpXDnEGUpggldlAKTT9
2024-09-28T19:43:39.604416Z ERROR process_queue_task{task_type="MessageV1"}: svix_server::worker: Unexpected: message doesn't exist msg_2mhUtDVDnpXDnEGUpggldlAKTT9
2024-09-28T19:44:24.887610Z ERROR process_queue_task{task_type="MessageV1"}: svix_server::worker: Unexpected: message doesn't exist msg_2mhUtDVDnpXDnEGUpggldlAKTT9
2024-09-28T19:45:10.163295Z ERROR process_queue_task{task_type="MessageV1"}: svix_server::worker: Unexpected: message doesn't exist msg_2mhUtDVDnpXDnEGUpggldlAKTT9
2024-09-28T19:45:55.445454Z ERROR process_queue_task{task_type="MessageV1"}: svix_server::worker: Unexpected: message doesn't exist msg_2mhUtDVDnpXDnEGUpggldlAKTT9
2024-09-28T19:46:40.710052Z ERROR process_queue_task{task_type="MessageV1"}: svix_server::worker: Unexpected: message doesn't exist msg_2mhUtDVDnpXDnEGUpggldlAKTT9

and it looks like, I will see this error forever. that´s why I need to know if there is a way to delete a msg from the queue ?
doesn´t play a role if I have to delete something directly in the database or redis as long I get rid of this error messages ;)

THANKS!

[michabbb]

for the record, a "delete message" is not working 😢

I guess that´s the problem, because the msg no longer exists in the database but somewhere in a queue....

[tasn]

There's no way to delete a message, this just deletes the payload. E.g. try to fetch the message and it should show.

[michabbb]

the msg does not exist in the database. so I guess the situation here is that the message is stuck between worker and redis somewhere, I don´t know... a "fetch msg" does not work, of course, because the msg does not exist in the DB... i will clear redis and do a restart... let´s see if that helps....

[tasn]

The message does exist, just not its content. Mind turning this into a ticket? We'll fix that asap.

[michabbb]

sure, do whatever you need. but just for the record, I wanted to get rid of this because this spams my logging....
so I did this:

docker exec -it <redis-container-name> redis-cli
FLUSHDB
FLUSHALL

and made a docker-compose restart

looks like this helped.... i don´t get the error msg any more.....

[tasn]

I'm not sure what's going on, deferring to the team.

[svix-james]

@michabbb Are you able to confirm whether the message in question exists in the database (i.e., in the message table)? I'm guessing from your previous message that may have already checked this, but I just want to confirm.

If it indeed does not exist in the database, do you have any idea how that might have happened? Under ordinary operation, I'm not aware of a situation that could cause this problem in the application.

[michabbb]

@svix-james the msg didn´t exist in the table, i checked that. i already fixed that as explained. but maybe you can replicate that by:

• create an endpoint with a none working url
• wait till there are 4 attempts
• delete the event-type of the message that couldn´t be delivered

I only can remember that I did something like that.
that all happened while I was testing the internal webhooks (message.attempt.failing).
and after 4 failed attempts I got my message.attempt.failing notification. but while
the message still was somewhere in the queue, I renamed or deleted the event-type
that was used with that message. between all that I restarted everything several times: docker-compose stop && docker-compose rm -f && docker-compose up -d

that´s the only thing I can remember, sorry.

[svix-james]

Thanks for the additional info. Let me see if I can come up w/ a repro.

[svix-james]

I'm not able to re-create this scenario and don't see any cases under which a message could be deleted from the database through the application. I think the most important thing to avoid the problem in the future is to add DLQ support for redis queues, which is already underway and should be available in the next couple of weeks.

[michabbb]

thanks for testing. whenever this happens again, i will let you know!

[tasn]

Closing this for now. If it comes up again please open a ticket and we can investigate! @svix-james is already adding some behavior improvements for this scenario. Stay tuned!