Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
For `outlines/vllm` previously FSM-sequence correspondence was broken, resulting FSM state being mixed between sequences, corrupting output. To alleviate this, we have `_patched_apply_logits_processor` which passes a stable sequence ID to the logits processor. In this PR we eliminate `_patched_apply_logits_processor` and cache FSM state based on the states input IDs. Continuation of #539 but much simpler because vllm upgrade fixed a lot of the issues being addressed there. Related discussions: - #624 Fixes: - Fixes #605 - Fixes #610 Already fixed: - #524 (this one can be closed, as it's was addressed previously by upgrading vllm) @viktor-ferenczi can you please confirm whether this branch fixes either #610 or #605 # Smoke tests ### basic parallel passed <details> ``` import json import vllm from pydantic import BaseModel from typing import List import torch import pandas as pd from outlines.serve.vllm import JSONLogitsProcessor class ConceptsList(BaseModel): concepts: List[str] BASE_MODEL = "microsoft/phi-2" llm = vllm.LLM(model=BASE_MODEL, tensor_parallel_size=1, dtype=torch.float16, max_model_len=2048) logits_processor = JSONLogitsProcessor(ConceptsList, llm.llm_engine) full_prompts = [ f"Provide me a list of {i} strings with key 'concepts'" for i in range(20) ] batch_results = llm.generate( full_prompts, sampling_params=vllm.SamplingParams( max_tokens=2048, logits_processors=[logits_processor] ), ) for result in batch_results: for output in result.outputs: json.loads(output.text) ``` </details> ### never ending regex passed <details> `python3 -m outlines.serve.serve --model="microsoft/phi-2"` ``` curl http://127.0.0.1:8000/generate \ -d '{ "prompt": "Sequence of numbers and letters:", "regex": "([123]-[abc]-([def]-)?)*", "n": 7 }' {"text":["Sequence of numbers and letters:1-a-1-b-1-c-1-a-","Sequence of numbers and letters:1-a-2-b-3-c-1-a-","Sequence of numbers and letters:1-a-2-b-3-c-d-1-","Sequence of numbers and letters:2-a-1-b-2-c-1-b-","Sequence of numbers and letters:2-b-3-c-d-2-b-3-","Sequence of numbers and letters:2-a-3-b-2-b-1-c-","Sequence of numbers and letters:2-a-3-b-d-2-a-3-"]} # rules for the above to validate correct FSM-sequence correspondence: # [123] always followed by [abc], [def] only ever preceded by [abc] # 1-a-1-b-1-c-1-a- # 1-a-2-b-3-c-1-a- # 1-a-2-b-3-c-d-1- # 2-a-1-b-2-c-1-b- # 2-b-3-c-d-2-b-3- # 2-a-3-b-2-b-1-c- # 2-a-3-b-d-2-a-3- ``` </details> ### sometimes ending early regex passed <details> `python3 -m outlines.serve.serve --model="microsoft/phi-2"` ``` curl http://127.0.0.1:8000/generate \ -d '{ "prompt": "Sequence of numbers and letters:", "regex": "([123]-[abc]-([def]-)?){3}", "n": 16 }' ``` output ``` {"text":["Sequence of numbers and letters:1-a-2-b-3-c-d-","Sequence of numbers and letters:1-a-2-b-3-c-d-","Sequence of numbers and letters:1-a-2-b-3-c-d-","Sequence of numbers and letters:1-a-2-b-3-c-d-","Sequence of numbers and letters:1-a-2-b-3-c-d-","Sequence of numbers and letters:3-a-1-b-2-c-d-","Sequence of numbers and letters:2-a-1-b-3-c-d-","Sequence of numbers and letters:1-a-1-b-1-c-d-","Sequence of numbers and letters:2-a-3-b-d-1-c-e-","Sequence of numbers and letters:1-b-3-a-2-c-d-","Sequence of numbers and letters:3-a-d-1-b-e-2-c-","Sequence of numbers and letters:1-a-3-b-1-b-d-","Sequence of numbers and letters:3-a-f-2-b-d-1-c-","Sequence of numbers and letters:1-b-d-3-a-e-2-c-","Sequence of numbers and letters:3-c-1-b-d-1-a-e-","Sequence of numbers and letters:1-c-1-c-e-1-b-e-"]} ``` analysis: ``` 1-a-2-b-3-c-d- 1-a-2-b-3-c-d- 1-a-2-b-3-c-d- 1-a-2-b-3-c-d- 1-a-2-b-3-c-d- 1-a-2-b-3-c-d- 3-a-1-b-2-c-d- 2-a-1-b-3-c-d- 1-a-1-b-1-c-d- 2-a-3-b-d-1-c-e- 1-b-3-a-2-c-d- 3-a-d-1-b-e-2-c- 1-a-3-b-1-b-d- 3-a-f-2-b-d-1-c- 1-b-d-3-a-e-2-c- 3-c-1-b-d-1-a-e- 1-c-1-c-e-1-b-e- ``` Observations: - All patterns are correct - Patterns don't "borrow" FSM state from one-another, they retain their own independent state - Some patterns produced more tokens than others successfully </details> ### Viktor's regex passed <details> `python3 -m outlines.serve.serve --model="microsoft/phi-2"` ``` curl http://127.0.0.1:8000/generate \ -d '{ "prompt": "You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer.\n\nYou are a helpful AI assistant. You give concise answers. If you do not know something, then say so.\n### Instruction:\nWrite down the first 10 prime numbers as a comma separated list, starting with 2.\n\n### Response:\n", "n": 1, "best_of": 1, "presence_penalty": 0.0, "frequency_penalty": 0.0, "repetition_penalty": 1.0, "temperature": 0.0, "top_p": 1.0, "top_k": -1, "min_p": 0.0, "use_beam_search": false, "length_penalty": 1.0, "early_stopping": false, "stop": [], "stop_token_ids": [], "include_stop_str_in_output": false, "ignore_eos": false, "max_tokens": 50, "logprobs": null, "prompt_logprobs": null, "skip_special_tokens": true, "spaces_between_special_tokens": true, "regex": "\\d+(\\s*,\\s*\\d+)*\\s*" }' ``` output: ``` {"text":["You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer.\n\nYou are a helpful AI assistant. You give concise answers. If you do not know something, then say so.\n### Instruction:\nWrite down the first 10 prime numbers as a comma separated list, starting with 2.\n\n### Response:\n2, 3, 5, 7, 11, 13, 17, 19, 23, 29\n"]} ``` </details> ### Viktors schema passed <details> `python3 -m outlines.serve.serve --model="microsoft/phi-2"` ``` curl http://127.0.0.1:8000/generate \ -d '{ "prompt": "You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer.\n\nYou are a helpful AI assistant. You give concise answers. If you do not know something, then say so.\n### Instruction:\nWrite a JSON describing a random fruit. It must conform to the following JSON schema: {\"properties\": {\"kind\": {\"title\": \"Kind\", \"type\": \"string\"}, \"color\": {\"title\": \"Color\", \"type\": \"string\"}, \"count\": {\"title\": \"Count\", \"type\": \"integer\"}, \"weight\": {\"title\": \"Weight\", \"type\": \"number\"}, \"sweet\": {\"title\": \"Sweet\", \"type\": \"boolean\"}}, \"required\": [\"kind\", \"color\", \"count\", \"weight\", \"sweet\"], \"title\": \"Fruit\", \"type\": \"object\"}\n\n### Response:\n", "n": 5, "best_of": 5, "presence_penalty": 0.0, "frequency_penalty": 0.0, "repetition_penalty": 1.0, "temperature": 1.0, "top_p": 1.0, "top_k": -1, "min_p": 0.0, "use_beam_search": false, "length_penalty": 1.0, "early_stopping": false, "stop": [], "stop_token_ids": [], "include_stop_str_in_output": false, "ignore_eos": false, "max_tokens": 200, "logprobs": null, "prompt_logprobs": null, "skip_special_tokens": true, "spaces_between_special_tokens": true, "schema": { "properties": { "kind": { "title": "Kind", "type": "string" }, "color": { "title": "Color", "type": "string" }, "count": { "title": "Count", "type": "integer" }, "weight": { "title": "Weight", "type": "number" }, "sweet": { "title": "Sweet", "type": "boolean" } }, "required": [ "kind", "color", "count", "weight", "sweet" ], "title": "Fruit", "type": "object" } }' ``` output: ``` {"text":["You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer.\n\nYou are a helpful AI assistant. You give concise answers. If you do not know something, then say so.\n### Instruction:\nWrite a JSON describing a random fruit. It must conform to the following JSON schema: {\"properties\": {\"kind\": {\"title\": \"Kind\", \"type\": \"string\"}, \"color\": {\"title\": \"Color\", \"type\": \"string\"}, \"count\": {\"title\": \"Count\", \"type\": \"integer\"}, \"weight\": {\"title\": \"Weight\", \"type\": \"number\"}, \"sweet\": {\"title\": \"Sweet\", \"type\": \"boolean\"}}, \"required\": [\"kind\", \"color\", \"count\", \"weight\", \"sweet\"], \"title\": \"Fruit\", \"type\": \"object\"}\n\n### Response:\n{\n\"kind\": \"Apple\",\n\"color\": \"Red\",\n\"count\": 10,\n\"weight\": 0.2,\n\"sweet\": true\n}","You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer.\n\nYou are a helpful AI assistant. You give concise answers. If you do not know something, then say so.\n### Instruction:\nWrite a JSON describing a random fruit. It must conform to the following JSON schema: {\"properties\": {\"kind\": {\"title\": \"Kind\", \"type\": \"string\"}, \"color\": {\"title\": \"Color\", \"type\": \"string\"}, \"count\": {\"title\": \"Count\", \"type\": \"integer\"}, \"weight\": {\"title\": \"Weight\", \"type\": \"number\"}, \"sweet\": {\"title\": \"Sweet\", \"type\": \"boolean\"}}, \"required\": [\"kind\", \"color\", \"count\", \"weight\", \"sweet\"], \"title\": \"Fruit\", \"type\": \"object\"}\n\n### Response:\n{\n \"kind\": \"Apple\",\n \"color\": \"Red\",\n \"count\": 10,\n \"weight\": 0.2,\n \"sweet\": true\n}","You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer.\n\nYou are a helpful AI assistant. You give concise answers. If you do not know something, then say so.\n### Instruction:\nWrite a JSON describing a random fruit. It must conform to the following JSON schema: {\"properties\": {\"kind\": {\"title\": \"Kind\", \"type\": \"string\"}, \"color\": {\"title\": \"Color\", \"type\": \"string\"}, \"count\": {\"title\": \"Count\", \"type\": \"integer\"}, \"weight\": {\"title\": \"Weight\", \"type\": \"number\"}, \"sweet\": {\"title\": \"Sweet\", \"type\": \"boolean\"}}, \"required\": [\"kind\", \"color\", \"count\", \"weight\", \"sweet\"], \"title\": \"Fruit\", \"type\": \"object\"}\n\n### Response:\n{\n \"kind\": \"apple\",\n \"color\": \"red\",\n \"count\": 5,\n \"weight\": 0.1,\n \"sweet\": true\n}","You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer.\n\nYou are a helpful AI assistant. You give concise answers. If you do not know something, then say so.\n### Instruction:\nWrite a JSON describing a random fruit. It must conform to the following JSON schema: {\"properties\": {\"kind\": {\"title\": \"Kind\", \"type\": \"string\"}, \"color\": {\"title\": \"Color\", \"type\": \"string\"}, \"count\": {\"title\": \"Count\", \"type\": \"integer\"}, \"weight\": {\"title\": \"Weight\", \"type\": \"number\"}, \"sweet\": {\"title\": \"Sweet\", \"type\": \"boolean\"}}, \"required\": [\"kind\", \"color\", \"count\", \"weight\", \"sweet\"], \"title\": \"Fruit\", \"type\": \"object\"}\n\n### Response:\n{\n\"kind\": \"Apple\",\n\"color\": \"Red\",\n\"count\": 10,\n\"weight\": 0.24,\n\"sweet\": true\n}","You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer.\n\nYou are a helpful AI assistant. You give concise answers. If you do not know something, then say so.\n### Instruction:\nWrite a JSON describing a random fruit. It must conform to the following JSON schema: {\"properties\": {\"kind\": {\"title\": \"Kind\", \"type\": \"string\"}, \"color\": {\"title\": \"Color\", \"type\": \"string\"}, \"count\": {\"title\": \"Count\", \"type\": \"integer\"}, \"weight\": {\"title\": \"Weight\", \"type\": \"number\"}, \"sweet\": {\"title\": \"Sweet\", \"type\": \"boolean\"}}, \"required\": [\"kind\", \"color\", \"count\", \"weight\", \"sweet\"], \"title\": \"Fruit\", \"type\": \"object\"}\n\n### Response:\n{\n \"kind\": \"Apple\",\n \"color\": \"red\",\n \"count\": 5,\n \"weight\": 0.3,\n \"sweet\": true\n}"]} ``` </details> --------- Co-authored-by: Andrew Lapp <[email protected]>
- Loading branch information
