How to download model from Huggingface and convert and use it in local machine?

[fingers10]

I was checking for example to use LlamaSharp with semantic kernel. But I'm not sure how to download the model and convert it and load it from modelPath. I'm not able to find steps on how to do this. Please can anyone assist me?

using LLama;
using LLama.Common;
using LLamaSharp.SemanticKernel.ChatCompletion;
using Microsoft.SemanticKernel.ChatCompletion;

Console.ForegroundColor = ConsoleColor.Yellow;

var modelPath = "";

// Load weights into memory
var parameters = new ModelParams(modelPath);
using var model = await LLamaWeights.LoadFromFileAsync(parameters);
var ex = new StatelessExecutor(model, parameters);

var chatGPT = new LLamaSharpChatCompletion(ex);

var chatHistory = chatGPT.CreateNewChat("This is a conversation between the " +
    "assistant and the user. \n\n You are a librarian, expert about books. ");

Console.WriteLine("Chat content:");
Console.WriteLine("------------------------");

chatHistory.AddUserMessage("Hi, I'm looking for book suggestions");
await MessageOutputAsync(chatHistory);

// First bot assistant message
var reply = await chatGPT.GetChatMessageContentAsync(chatHistory);
chatHistory.AddAssistantMessage(reply.Content);
await MessageOutputAsync(chatHistory);

// Second user message
chatHistory.AddUserMessage("I love history and philosophy, I'd like to learn " +
    "something new about Greece, any suggestion");
await MessageOutputAsync(chatHistory);

// Second bot assistant message
reply = await chatGPT.GetChatMessageContentAsync(chatHistory);
chatHistory.AddAssistantMessage(reply.Content);
await MessageOutputAsync(chatHistory);
return;

/// <summary>
/// Outputs the last message of the chat history
/// </summary>
static Task MessageOutputAsync(Microsoft.SemanticKernel.ChatCompletion.ChatHistory chatHistory)
{
    var message = chatHistory.Last();

    Console.WriteLine($"{message.Role}: {message.Content}");
    Console.WriteLine("------------------------");

    return Task.CompletedTask;
}

[martindevans]

You need a gguf model, most models are already converted and published online. bartowski is well known for converting many models.

If you want to conver it yourself I don't know much about that process, docs for it can be found on the llama.cpp repo here.

[fingers10]

You need a gguf model, most models are already converted and published online

@martindevans where can I find the already converted ones?

[martindevans]

Bartowski (linked previously) has 1152 converted GGUF models. If he doesn't have it, just searching HuggingFace for "GGUF" will find even more.