You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What behavior of the library made you think about the improvement?
In #956 I noticed some performance issues relating to a large number of allowed_tokens. We should profile how long it takes for a logits processor to augment logits:
The core performance issue with outlines.generate.regex(model, ".{200}") is the need to convert a large (~150,000 integer) list into a tensor in the logits processor
To mitigate, we can create a separate issue to ensure the FSM index uses tensors of token IDs, not lists. This will result in self.fsm.get_next_instruction(self._fsm_state).tokens being a tensor of token IDs.
How would you like it to behave?
As part of our ASV benchmark test suite, we should benchmark the performance of OutlinesLogitsProcessor to ensure there aren't performance regressions, and track performance improvements.
The text was updated successfully, but these errors were encountered:
What behavior of the library made you think about the improvement?
In #956 I noticed some performance issues relating to a large number of
allowed_tokens
. We should profile how long it takes for a logits processor to augment logits:How would you like it to behave?
As part of our ASV benchmark test suite, we should benchmark the performance of
OutlinesLogitsProcessor
to ensure there aren't performance regressions, and track performance improvements.The text was updated successfully, but these errors were encountered: