Model: Log metrics before yielding a stop

Yielding the finish reason before the logging causes the function to terminate early. Instead, log before yielding and breaking out of the generation loop. Signed-off-by: kingbri <[email protected]>
theroyallab · Mar 20, 2024 · b74603d · b74603d
1 parent 09a4c79
commit b74603d
Showing 1 changed file with 11 additions and 11 deletions.
diff --git a/backends/exllamav2/model.py b/backends/exllamav2/model.py
@@ -975,19 +975,19 @@ def generate_gen_sync(self, prompt: str, **kwargs):
                 last_chunk_time = now
 
             if eos or generated_tokens == max_tokens:
+                # Print response
+                log_response(full_response)
+
+                # Print metrics
+                elapsed_time = last_chunk_time - start_time
+                context_len = None if ids is None else context_len
+
+                log_metrics(
+                    generated_tokens, elapsed_time, context_len, self.config.max_seq_len
+                )
+
                 finish_reason = "length" if generated_tokens == max_tokens else "stop"
                 generation = {"finish_reason": finish_reason}
                 yield generation
 
                 break
-
-        # Print response
-        log_response(full_response)
-
-        # Print metrics
-        elapsed_time = last_chunk_time - start_time
-        context_len = None if ids is None else context_len
-
-        log_metrics(
-            generated_tokens, elapsed_time, context_len, self.config.max_seq_len
-        )