Red5 CPU usage 100%

[anvishwakarma]

Issue

The red5 threads enter in infinite loop and consumes 100% cpu.

Short description

When RTMP client disconnects, it is not able to clear the session buffer and while trying to do so it enters in infinite loop due to incorrect logic in RTMPProtocolDecoder.java (decodebuffer).

Environment

Operating system and version: Ubuntu 16.04
Java version: java 8
Red5 version: 1.0.9

Expected behavior

Session should be disconnected and all resources released.

Actual behavior

Session disconnects but enters in infinite loop

[mondain]

How do you arrive at this conclusion and what are your steps to reproduce the issue?

[anvishwakarma]

I rebuilt the red5 from source code and and added the logs to confirm. After that I modified this file (RTMPProtocolDecoder.java) to avoid the infinite loop which fixed my problem.

[mondain]

@anvishwakarma would you mind posting what you changed and where this supposed infinite loop is located? Also have you tested the latest pre-release? https://github.com/Red5/red5-server/releases/tag/v1.0.10-M9

[anvishwakarma]

In the decodeBuffer method (Line#102), it enters in infinite loop (Line#123) as buffer is never cleared in case of error. Therefore, I have to insert following line of code after Line#136.
else if(conn.getStateCode() == RTMP.STATE_ERROR || conn.getStateCode() == RTMP.STATE_DISCONNECTING || conn.getStateCode() == RTMP.STATE_DISCONNECTED){ buffer.clear(); break; }
We have not tried the latest pre-release https://github.com/Red5/red5-server/releases/tag/v1.0.10-M9. But we are planning as we have facing lots of CLOSED_WAIT issues as well.
Do you have any suggestions for CLOSED_WAIT issue?

[anvishwakarma]

We tried with release 1.0.10-M9. But it has the same issue.