process and track sync that started before the log was created

[Fujio-Turner]

The log currently only cares about new syncs or log lines that have /_blipsync because this line is only real line that tells you who(sync gateway user Id) is creating a replication.

So you might run the scritp and catch lets say 30% of the users , but what about the other 70% who is "headless" no known user name , but is sync-ing or have a open websocket with same _changes and /or data pushed or pulled down.

I need to create a foo user like "ichabod:{websocket ID}"

Its not going to have an incomplete set of information b/c you don't know when the replication began.

But I need to know the full picture of how many users/replications are in the logs , even If I don't know who and when they started.

[Fujio-Turner]

similar to #9

[Fujio-Turner]

To address the challenge of tracking WebSocket connections that started before the log file's time frame, you can implement a strategy to:

1. Identify Existing Connections: Detect and track WebSocket connections that were already open when the log started.
2. Track New Connections: Continue to track new connections as they are established.

Here's how you might approach this:

1. Detecting Pre-existing Connections

• Log Analysis: When you start processing a log, look for any WebSocket activity that indicates an open connection. This might involve:

  • Checking for lines where a WebSocket connection is already in use without a preceding connection establishment.
  • Using a timestamp comparison to infer if a connection was established before the log's start time.

2. Tracking All Connections

• WebSocket ID Tracking: Maintain a set or dictionary of WebSocket IDs:

  class SGLogReader:
      def __init__(self):
          self.websocket_connections = set()
          self.websocket_start_times = {}  # To track when each WebSocket was first seen
  
      def process_log_line(self, line):
          # Parse the line to check for WebSocket activity
          if 'WebSocket' in line and '[#' in line:
              ws_id = self.extract_websocket_id(line)
              if ws_id not in self.websocket_connections:
                  self.websocket_connections.add(ws_id)
                  self.websocket_start_times[ws_id] = self.get_current_time()  # Assuming you have a method to get the current time or line's timestamp
  
          # Other processing logic
  
      def extract_websocket_id(self, line):
          # Extract the WebSocket ID from the log line
          # This might involve regex or string manipulation
          pass
  
      def get_current_time(self):
          # Method to get the current time or line's timestamp
          pass

3. Handling Pre-existing Connections

• Assumption: If a WebSocket ID appears without a prior connection event, assume it's pre-existing:

  def process_log(self):
      for line in self.log_file:
          if 'WebSocket' in line and '[#' in line:
              ws_id = self.extract_websocket_id(line)
              if ws_id not in self.websocket_connections:
                  # This WebSocket ID wasn't seen before, might be pre-existing
                  self.websocket_connections.add(ws_id)
                  # Optionally, you might want to mark this as pre-existing or log it for further analysis
                  self.websocket_start_times[ws_id] = "pre-existing"
          # Continue with other processing

4. Counting and Reporting

• Counting: At the end of log processing, you can