fix: handle invalid json issue

[1abhishekpandey]

Description

• We are addressing the invalid JSON issue by filtering out such messages during batch construction and ensuring their removal from the database.
• Our hypothesis is that the invalid message string is somehow a single character, based on the sample invalid message payloads we have observed.

Implementation details

• To resolve this issue, we need to filter the invalid events from the database.
• First, we need to prevent the construction of an invalid batch payload by detecting the invalid JSON:
  • Filtering invalid events during batch construction: For each message, we check whether it is a single character (based on our hypothesis). If it is, we treat it as invalid JSON and exclude it from the batch. If it is a valid event, we construct a proper batch of events. The goal is to ensure that the batch does not contain any invalid events. We construct batches in the FlushUtils class.
• Second, we need to remove any invalid messages from the database:
  • To remove events from the database, we need to mark them as completed for both device and cloud modes.
    • Device: If events are not marked as device mode done and are invalid, during SDK initialization in RudderDeviceModeManager.replayMessageQueue(), we mark all events that are null (note: if there is an issue during deserialization, the message becomes null) as device mode processing done.
    • Cloud: If events are not marked as cloud mode done, we need to handle multiple scenarios:
      • At CloudModeProcessor:
        • If there is a single message and is invalid, the batch payload will be null. In this case, we need to directly mark the event as cloud mode processing done.
        • If there is more than one event in the batch then after sending the filtered batch, we mark all the events as device mode processing done, including the invalid message and eventually remove them from the DB.
      • At FlushUtils: This class handles manual flush:
        • For manual flush: Similar to CloudModeProcessor, we mark events as cloud mode processing done by handling both scenarios: when only a single event is invalid, or when there is a filtered batch.

NOTE: Our fix, based on the hypothesis, will not affect the existing SDK behaviour. We have targeted it to handle only a specific error. If there are other types of errors that are not yet known, we cannot address them with this fix, and they will require further research, including a sample invalid JSON payloads made using the latest version of the SDK.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist:

• Version upgraded (project, README, gradle, podspec etc)
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added unit tests for the code
• I have made corresponding changes to the documentation

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud