feat: support advanced non-incremental output

[kevinlin09]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @kevinlin09, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly improves the handling of non-incremental output in streaming calls for both 'Generation' and 'MultiModalConversation' APIs. It introduces an intelligent internal merging strategy that reconstructs complete responses, including complex tool call structures, from incremental streams. This allows users to request a single, consolidated output even when the underlying API streams data, enhancing usability and consistency.

Highlights

• Enhanced Non-Incremental Streaming: Implemented a mechanism to simulate non-incremental output for streaming API calls by internally enabling incremental output and then merging the streamed chunks, ensuring a complete response is delivered at once.
• Robust Tool Call Merging: Introduced sophisticated logic to correctly accumulate and merge partial tool call definitions (including function names and arguments) received across multiple streaming responses.
• Model-Specific Streaming Control: Added conditional logic to determine which models should have their 'incrementalOutput' parameter internally overridden for this merging behavior, excluding specific models like "tts" or "omni".
• Updated Sample Code: Provided new examples in 'GenerationStreamCall.java' and 'MultiModalConversationQwenVL.java' demonstrating the use of tool calls with streaming and the non-incremental output feature.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces support for non-incremental streaming output, which is particularly useful for features like tool calls. The implementation cleverly simulates this by internally using incremental streaming and accumulating the results. The changes are mostly in Generation.java and MultiModalConversation.java, with corresponding updates to sample files. My review focuses on the core merging logic, where I've identified a critical bug, some areas for performance improvement, and a potential issue with future extensibility. There is also significant code duplication between Generation.java and MultiModalConversation.java that should be addressed by refactoring the common logic into a shared component to improve maintainability.

[gemini-code-assist]

The logic for merging tool calls is flawed. If a tool call with a given index already exists in accumulatedToolCalls but is not of type ToolCallFunction, the else branch is taken, which incorrectly adds a new tool call with the same index. This leads to duplicate tool calls for the same index in the accumulated list.

The logic should be restructured to first check for the existence of a tool call with the given index, and then decide whether to merge with it (if it exists) or add a new one (if it doesn't).

      if (existingCall != null) {
        if (existingCall instanceof ToolCallFunction &&
                currentCall instanceof ToolCallFunction) {
          // Merge function calls
          ToolCallFunction existingFunctionCall = (ToolCallFunction) existingCall;
          ToolCallFunction currentFunctionCall = (ToolCallFunction) currentCall;

          if (currentFunctionCall.getFunction() != null) {
            // Ensure existing function call has a function object
            if (existingFunctionCall.getFunction() == null) {
              existingFunctionCall.setFunction(existingFunctionCall.new CallFunction());
            }

            // Accumulate arguments if present
            if (currentFunctionCall.getFunction().getArguments() != null) {
              String existingArguments = existingFunctionCall.getFunction().getArguments();
              if (existingArguments == null) {
                existingArguments = "";
              }
              String currentArguments = currentFunctionCall.getFunction().getArguments();
              existingFunctionCall.getFunction().setArguments(existingArguments + currentArguments);
            }

            // Accumulate function name if present
            if (currentFunctionCall.getFunction().getName() != null) {
              String existingName = existingFunctionCall.getFunction().getName();
              if (existingName == null) {
                existingName = "";
              }
              String currentName = currentFunctionCall.getFunction().getName();
              existingFunctionCall.getFunction().setName(existingName + currentName);
            }

            // Update function output if present
            if (currentFunctionCall.getFunction().getOutput() != null) {
              existingFunctionCall.getFunction().setOutput(currentFunctionCall.getFunction().getOutput());
            }
          }

          // Update other fields with latest non-empty values
          if (currentFunctionCall.getIndex() != null) {
            existingFunctionCall.setIndex(currentFunctionCall.getIndex());
          }
          if (currentFunctionCall.getId() != null && !currentFunctionCall.getId().isEmpty()) {
            existingFunctionCall.setId(currentFunctionCall.getId());
          }
          if (currentFunctionCall.getType() != null) {
            existingFunctionCall.setType(currentFunctionCall.getType());
          }
        }
      } else {
        // Add new tool call (create a copy)
        if (currentCall instanceof ToolCallFunction) {
          ToolCallFunction currentFunctionCall = (ToolCallFunction) currentCall;
          ToolCallFunction newFunctionCall = new ToolCallFunction();
          newFunctionCall.setIndex(currentFunctionCall.getIndex());
          newFunctionCall.setId(currentFunctionCall.getId());
          newFunctionCall.setType(currentFunctionCall.getType());

          if (currentFunctionCall.getFunction() != null) {
            ToolCallFunction.CallFunction newCallFunction = newFunctionCall.new CallFunction();
            newCallFunction.setName(currentFunctionCall.getFunction().getName());
            newCallFunction.setArguments(currentFunctionCall.getFunction().getArguments());
            newCallFunction.setOutput(currentFunctionCall.getFunction().getOutput());
            newFunctionCall.setFunction(newCallFunction);
          }

          accumulatedToolCalls.add(newFunctionCall);
        } else {
          // For other types of tool calls, add directly (assuming they are immutable or don't need merging)
          accumulatedToolCalls.add(currentCall);
        }
      }

[gemini-code-assist]

The logic for merging tool calls is flawed. If a tool call with a given index already exists in accumulatedToolCalls but is not of type ToolCallFunction, the else branch is taken, which incorrectly adds a new tool call with the same index. This leads to duplicate tool calls for the same index in the accumulated list.

The logic should be restructured to first check for the existence of a tool call with the given index, and then decide whether to merge with it (if it exists) or add a new one (if it doesn't).

      if (existingCall != null) {
        if (existingCall instanceof ToolCallFunction &&
                currentCall instanceof ToolCallFunction) {
          // Merge function calls
          ToolCallFunction existingFunctionCall = (ToolCallFunction) existingCall;
          ToolCallFunction currentFunctionCall = (ToolCallFunction) currentCall;

          if (currentFunctionCall.getFunction() != null) {
            // Ensure existing function call has a function object
            if (existingFunctionCall.getFunction() == null) {
              existingFunctionCall.setFunction(existingFunctionCall.new CallFunction());
            }

            // Accumulate arguments if present
            if (currentFunctionCall.getFunction().getArguments() != null) {
              String existingArguments = existingFunctionCall.getFunction().getArguments();
              if (existingArguments == null) {
                existingArguments = "";
              }
              String currentArguments = currentFunctionCall.getFunction().getArguments();
              existingFunctionCall.getFunction().setArguments(existingArguments + currentArguments);
            }

            // Accumulate function name if present
            if (currentFunctionCall.getFunction().getName() != null) {
              String existingName = existingFunctionCall.getFunction().getName();
              if (existingName == null) {
                existingName = "";
              }
              String currentName = currentFunctionCall.getFunction().getName();
              existingFunctionCall.getFunction().setName(existingName + currentName);
            }

            // Update function output if present
            if (currentFunctionCall.getFunction().getOutput() != null) {
              existingFunctionCall.getFunction().setOutput(currentFunctionCall.getFunction().getOutput());
            }
          }

          // Update other fields with latest non-empty values
          if (currentFunctionCall.getIndex() != null) {
            existingFunctionCall.setIndex(currentFunctionCall.getIndex());
          }
          if (currentFunctionCall.getId() != null && !currentFunctionCall.getId().isEmpty()) {
            existingFunctionCall.setId(currentFunctionCall.getId());
          }
          if (currentFunctionCall.getType() != null) {
            existingFunctionCall.setType(currentFunctionCall.getType());
          }
        }
      } else {
        // Add new tool call (create a copy)
        if (currentCall instanceof ToolCallFunction) {
          ToolCallFunction currentFunctionCall = (ToolCallFunction) currentCall;
          ToolCallFunction newFunctionCall = new ToolCallFunction();
          newFunctionCall.setIndex(currentFunctionCall.getIndex());
          newFunctionCall.setId(currentFunctionCall.getId());
          newFunctionCall.setType(currentFunctionCall.getType());

          if (currentFunctionCall.getFunction() != null) {
            ToolCallFunction.CallFunction newCallFunction = newFunctionCall.new CallFunction();
            newCallFunction.setName(currentFunctionCall.getFunction().getName());
            newCallFunction.setArguments(currentFunctionCall.getFunction().getArguments());
            newCallFunction.setOutput(currentFunctionCall.getFunction().getOutput());
            newFunctionCall.setFunction(newCallFunction);
          }

          accumulatedToolCalls.add(newFunctionCall);
        } else {
          // For other types of tool calls, add directly (assuming they are immutable or don't need merging)
          accumulatedToolCalls.add(currentCall);
        }
      }

[gemini-code-assist]

The current implementation uses a linear search to find an existing tool call by its index within the accumulatedToolCalls list. This is inefficient, with a time complexity that can be problematic if the number of tool calls is large.

For better performance, consider changing accumulatedToolCalls from a List<ToolCallBase> to a Map<Integer, ToolCallBase> in the AccumulatedData class. Using the tool call index as the key would allow for O(1) average time complexity for lookups. This would also simplify the mergeToolCalls logic.

Example of the proposed change:

// In AccumulatedData class
private static class AccumulatedData {
  StringBuilder content = new StringBuilder();
  // Use a Map for efficient lookups by index.
  // Use LinkedHashMap to preserve insertion order if needed.
  Map<Integer, ToolCallBase> toolCalls = new LinkedHashMap<>();
}

// In mergeToolCalls method
private void mergeToolCalls(List<ToolCallBase> currentToolCalls, Map<Integer, ToolCallBase> accumulatedToolCalls) {
    for (ToolCallBase currentCall : currentToolCalls) {
        // ...
        int index = currentCall.getIndex();
        ToolCallBase existingCall = accumulatedToolCalls.get(index);
        if (existingCall != null) {
            // ... merge logic ...
        } else {
            // ... add new logic ...
            accumulatedToolCalls.put(index, newCall);
        }
    }
}

[gemini-code-assist]

When adding a new tool call that is not a ToolCallFunction, the currentCall object is added directly to the accumulatedToolCalls list. The comment mentions an assumption that other tool call types are immutable. This is a risky assumption and makes the code less robust for future extensions. If a new, mutable ToolCallBase subclass is introduced, this could lead to unintended side effects where the object in the stream is modified.

It would be safer to enforce a copy/clone mechanism for all ToolCallBase types to ensure that the accumulated data is always a deep copy.

[gemini-code-assist]

The current implementation uses a linear search to find an existing tool call by its index within the accumulatedToolCalls list. This is inefficient, with a time complexity that can be problematic if the number of tool calls is large.

For better performance, consider changing accumulatedToolCalls from a List<ToolCallBase> to a Map<Integer, ToolCallBase> in the AccumulatedData class. Using the tool call index as the key would allow for O(1) average time complexity for lookups. This would also simplify the mergeToolCalls logic.

Example of the proposed change:

// In AccumulatedData class
private static class AccumulatedData {
  List<Map<String, Object>> content = new ArrayList<>();
  // Use a Map for efficient lookups by index.
  // Use LinkedHashMap to preserve insertion order if needed.
  Map<Integer, ToolCallBase> toolCalls = new LinkedHashMap<>();
}

// In mergeToolCalls method
private void mergeToolCalls(List<ToolCallBase> currentToolCalls, Map<Integer, ToolCallBase> accumulatedToolCalls) {
    for (ToolCallBase currentCall : currentToolCalls) {
        // ...
        int index = currentCall.getIndex();
        ToolCallBase existingCall = accumulatedToolCalls.get(index);
        if (existingCall != null) {
            // ... merge logic ...
        } else {
            // ... add new logic ...
            accumulatedToolCalls.put(index, newCall);
        }
    }
}

[gemini-code-assist]

When adding a new tool call that is not a ToolCallFunction, the currentCall object is added directly to the accumulatedToolCalls list. The comment mentions an assumption that other tool call types are immutable. This is a risky assumption and makes the code less robust for future extensions. If a new, mutable ToolCallBase subclass is introduced, this could lead to unintended side effects where the object in the stream is modified.

It would be safer to enforce a copy/clone mechanism for all ToolCallBase types to ensure that the accumulated data is always a deep copy.