Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use outlines.processors for models.llamacpp #997

Merged
merged 2 commits into from
Jul 15, 2024

Conversation

lapp0
Copy link
Contributor

@lapp0 lapp0 commented Jun 21, 2024

Fixes #965

Problem

in main

  • generate.fsm only supports SequenceGenerator,
  • models.llamacpp doesn't support SequenceGenerator, it only supports SequenceGeneratorAdapter
  • integrations.llamacpp doesn't have a FSM logits processor.

Solution

  • Ensure models.llamacpp uses outlines.processors / SequenceGeneratorAdapter for all generators
  • Update generate.fsm to use SequenceGeneratorAdapter for all unified models
  • test_generate.py tests for generate.fsm on MLXLM, llamacpp, and transformers (the three models using outlines.processors)
    • fix critical generate.fsm bug discovered through this test in guide.py impacting all models

@lapp0 lapp0 force-pushed the fix-llamacpp-fsm branch 2 times, most recently from 3b4f1e6 to 7b8ab97 Compare July 15, 2024 08:10
@lapp0 lapp0 marked this pull request as ready for review July 15, 2024 08:55
@rlouf rlouf merged commit 5a7f082 into dottxt-ai:main Jul 15, 2024
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

LlamaCpp doesnt work with generate.fsm for custom FSMs
2 participants