Mixtral Mixture of Experts example #3075
Conversation
| logger.info("Model %s loading tokenizer", ctx.model_name) | ||
| self.model = AutoModelForCausalLM.from_pretrained( | ||
| model_path, | ||
| device_map="balanced", |
There was a problem hiding this comment.
according to the link, it seems "auto" can cover the future changes. Or you can allow cx to define in the model-config.yaml.
Quoted from the link:
"The options "auto" and "balanced" produce the same results for now, but the behavior of "auto" might change in the future if we find a strategy that makes more sense, while "balanced" will stay stable."
| - `low_cpu_mem_usage=True` for loading with limited resource using `accelerate` | ||
| - 8-bit quantization using `bitsandbytes` | ||
| - `Accelerated Transformers` using `optimum` | ||
| - TorchServe streaming response |
There was a problem hiding this comment.
Is this example going to demo microbatch+streaming or continuousbatching+streaming?
| ) | ||
| self.model.resize_token_embeddings(self.model.config.vocab_size + 1) | ||
|
|
||
| self.output_streamer = TextIteratorStreamerBatch( |
There was a problem hiding this comment.
you can use TS customized TextIteratorStreamerBatch with microbatching. See inf2 example:
| input_ids, attention_mask = self.encode_input_text(input_text["prompt"]) | ||
| input_ids_batch.append(input_ids) | ||
| attention_mask_batch.append(attention_mask) | ||
| params.append(input_text["params"]) |
There was a problem hiding this comment.
We are going to use openAI payload style (see https://github.com/lxning/benchmark-locust/blob/ts/llm_bench/load_test.py#L254) since openAI api is most popular. the params is flatten.
| self.output_streamer = TextIteratorStreamerBatch( | ||
| self.tokenizer, | ||
| batch_size=len(input_ids_batch), | ||
| skip_special_tokens=True, | ||
| ) | ||
| generation_kwargs = dict( | ||
| inputs=input_ids_batch, | ||
| attention_mask=attention_mask_batch, | ||
| streamer=self.output_streamer, | ||
| max_new_tokens=params[0]["max_new_tokens"], | ||
| temperature=params[0]["temperature"], | ||
| top_p=params[0]["top_p"], | ||
| ) | ||
| thread = Thread(target=self.model.generate, kwargs=generation_kwargs) | ||
| thread.start() | ||
|
|
||
| for new_text in self.output_streamer: | ||
| send_intermediate_predict_response( |
There was a problem hiding this comment.
you can check inf2 example to update this section to combine microbatch+streaming.
Description
This example shows how to deploy Mixtral-8x7B model with HuggingFace with the following features
low_cpu_mem_usage=Truefor loading with limited resource usingacceleratebitsandbytesAccelerated Transformersusingoptimumproduces the output
Fixes #(issue)
Type of change
Please delete options that are not relevant.
Feature/Issue validation/testing
Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.
Test A
Logs for Test A
Test B
Logs for Test B
Checklist: