[Filitered Stream] Stream freezes periodically.

[Clashkid155]

I noticed an annoying issue with streams where everything freezes with no error.
My code is the same as what's in #158.

The only log I can provide

Nov 03 06:19:09 SteamVul stream[79810]: Stream started.
Nov 03 06:22:15 SteamVul stream[79810]: remaining 297 reset: 1667457112
Nov 04 00:23:40 SteamVul stream[79810]: remaining 299 reset: 1667522315
Nov 04 08:47:34 SteamVul stream[79810]: remaining 299 reset: 1667552549
Nov 04 19:45:14 SteamVul stream[79810]: remaining 299 reset: 1667592008
Nov 04 19:48:30 SteamVul stream[79810]: remaining 298 reset: 1667592008
Nov 04 19:55:02 SteamVul stream[79810]: remaining 297 reset: 1667592008
Nov 05 10:12:43 SteamVul stream[79810]: remaining 299 reset: 1667644056
Nov 05 14:43:05 SteamVul stream[79810]: remaining 299 reset: 1667660280
Nov 05 14:43:39 SteamVul stream[79810]: remaining 298 reset: 1667660280
Nov 05 14:50:58 SteamVul stream[79810]: remaining 297 reset: 1667660280
Nov 05 15:32:07 SteamVul stream[79810]: remaining 299 reset: 1667663221
Nov 05 15:36:34 SteamVul stream[79810]: remaining 298 reset: 1667663221
Nov 05 15:36:38 SteamVul stream[79810]: remaining 297 reset: 1667663221
Nov 07 20:40:23 SteamVul stream[79810]: remaining 299 reset: 1667854518
Nov 07 20:40:46 SteamVul stream[79810]: remaining 298 reset: 1667854518
Nov 09 23:14:09 SteamVul systemd[1]: Stopping Twicker bot (Twitter & Firestore)...
Nov 09 23:14:09 SteamVul stream[79810]: Stream ended.  Cleaned  0 Purged  0
Nov 09 23:14:09 SteamVul systemd[1]: st-stream.service: Deactivated successfully.
Nov 09 23:14:09 SteamVul systemd[1]: Stopped Twicker bot (Twitter & Firestore).
Nov 09 23:14:09 SteamVul systemd[1]: st-stream.service: Consumed 2d 48min 50.846s CPU time.
Nov 09 23:14:09 SteamVul systemd[1]: Started Twicker bot (Twitter & Firestore).
Nov 09 23:14:09 SteamVul stream[183866]: Stream started.
Nov 09 23:14:56 SteamVul stream[183866]: remaining 299 reset: 1668036590

Each time this happens I've to reboot my service (systemd) which you can tell from the log.
So is there a way to keep checking if the stream is alive? I hope you can imagine how difficult it is to have this problem.

[Clashkid155]

@g8rswimmer Could you check this out?

[g8rswimmer]

@Clashkid155 Have you run with the latest release? I am wondering if the stream fix in the v2.1.5 release may have fixed this.

[Clashkid155]

@g8rswimmer v2.1.5 doesn't fix it.

I attached images of a recent tweet and bot log.

As you can see, the bot hasn't been active for 2 days, with no error whatsoever.

I think the stream is being killed, but never restarts.

[Clashkid155]

Just noticed something else, i can start another stream even when one is already running, this shouldn't happen since i don't have higher privilege (academic i think)

[g8rswimmer]

@Clashkid155
I think that the streaming is a HTTP connection that is long lived, if the stream is being killed, it would the the responsibility of the caller to restart it

[Clashkid155]

That's where the whole issue is, the caller doesn't know when it's been killed.

This doesn't trigger when the stream is killed

[g8rswimmer]

Did you use this channel for disconnection errors
https://github.com/g8rswimmer/go-twitter/blob/master/v2/tweet_stream.go#L469

[Clashkid155]

I haven't tried that, but don't you think the TweetStream.Conn... should also work when it disconnects?

Btw, Wasn't that introduced recently?

[g8rswimmer]

@Clashkid155
I ran a little test against the volume stream to see if I could get the disconnect to return false

I ran this example

It ran for about 25 hours but I did get the disconnect message here

I know that you are using the filter stream and I was wondering two things

1. can you provide the filter setup that you are using to reproduce
2. can you try and run the volume stream and see if you get the disconnected message?

[Clashkid155]

1. This

2. I'll try.

I think I've seen my error (no default in my select).
Will run so checks and post any update.

[Clashkid155]

@g8rswimmer Seems the whole issue was a user error sorry for wasting your time.
Btw, is there are method you recommend for restarting the stream?

[imax9000]

I think I've seen my error (no default in my select).

Having default: clause there means that your program will constantly consume 100% of a CPU core, even if it's doing nothing.

[Clashkid155]

I noticed that.
Maybe that's why i removed it at first