lets callers inject a prebuilt Tokenizer in the LanguageModel #278
Conversation
| var repetitionPenalty: Float? | ||
|
|
||
| @Option(help: "Path to a local folder containing tokenizer_config.json and tokenizer.json") | ||
| var tokenizerFolder: String? |
There was a problem hiding this comment.
| var tokenizerFolder: String? | |
| var tokenizerPath: String? |
(nit: this is perhaps more idiomatic in Swift APIs)
|
|
||
| ```swift | ||
| let compiledURL: URL = ... // path to .mlmodelc | ||
| let tokenizerFolder: URL = ... // folder containing tokenizer_config.json and tokenizer.json |
There was a problem hiding this comment.
| let tokenizerFolder: URL = ... // folder containing tokenizer_config.json and tokenizer.json | |
| let tokenizerURL: URL = ... // folder containing tokenizer_config.json and tokenizer.json |
| ) | ||
| ``` | ||
|
|
||
| Make sure the tokenizer assets come from the same Hugging Face repo as the original checkpoint. For the |
There was a problem hiding this comment.
| Make sure the tokenizer assets come from the same Hugging Face repo as the original checkpoint. For the | |
| Make sure the tokenizer assets come from the same Hugging Face repo as the original checkpoint or are compatible with the model you use. For the |
| if let configuration { | ||
| self.configuration = configuration | ||
| } else if tokenizer == nil { | ||
| self.configuration = LanguageModelConfigurationFromHub(modelName: modelName) | ||
| } else { | ||
| self.configuration = nil | ||
| } |
There was a problem hiding this comment.
I find it a bit confusing that if configuration is provided, then tokenizer will be silently ignored. These look like two different ways to inject a tokenizer. Could we maybe use multiple initializers instead?
Another option is to just remove the configuration argument for now and discuss in a new PR. Is the main reason to add it to provide a custom HubApi? That's useful, of course, but perhaps we could just provide that instead of the full configuration.
| tokenizerFolder: URL, | ||
| computeUnits: MLComputeUnits = .cpuAndGPU |
There was a problem hiding this comment.
| tokenizerFolder: URL, | |
| computeUnits: MLComputeUnits = .cpuAndGPU | |
| computeUnits: MLComputeUnits = .cpuAndGPU, | |
| tokenizer tokenizerFolder: URL, |
Making it look like an overloaded version of the previous method (keeping same order and overloading the type for tokenizer, while still using tokenizerFolder inside. This is usual in Swift APIs (although perhaps the tokenizer name could be somewhat misleading).
| tokenizer: Tokenizer, | ||
| computeUnits: MLComputeUnits = .cpuAndGPU |
There was a problem hiding this comment.
| tokenizer: Tokenizer, | |
| computeUnits: MLComputeUnits = .cpuAndGPU | |
| computeUnits: MLComputeUnits = .cpuAndGPU, | |
| tokenizer: Tokenizer, |
|
thanks! will look shortly and fix |
fixes #238