High-speed Large Language Model Serving on PCs with Consumer-grade GPUs
-
Updated
Sep 6, 2024 - C++
#
local-inference
Here are 7 public repositories matching this topic...
Fast Inference of MoE Models with CPU-GPU Orchestration
-
Updated
Nov 18, 2024 - Python
Tool for test diferents large language models without code.
api-rest chat-application flask-api inference-api large-language-models llm local-inference llamacpp llm-inference llama2 llama-cpp-python llama2-7b mixtral-8x7b
-
Updated
Sep 17, 2024 - JavaScript
LLM chatbot example using OpenVINO with RAG (Retrieval Augmented Generation).
natural-language-processing offline chatbot intel edge-computing rag openvino huggingface edge-inference cloud-free llm local-inference langchain dolly2 retrieval-augmented-generation llama2 neural-chat
-
Updated
Jan 25, 2024 - Python
Script which takes a .wav audio file, performs speech-to-text using OpenAI/Whisper, and then, using Llama3, summarization and action point from the transcript generated
summarization speech-to-text whisper meeting-minutes local-inference langchain-python llm-inference ollama
-
Updated
Sep 16, 2024 - Python
script which performs RAG and use a local LLM for Q&A
-
Updated
Sep 16, 2024 - Python
Nexa SDK is a comprehensive toolkit for supporting ONNX and GGML models. It supports text generation, image generation, vision-language models (VLM), auto-speech-recognition (ASR), and text-to-speech (TTS) capabilities.
-
Updated
Nov 19, 2024 - Python
Improve this page
Add a description, image, and links to the local-inference topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the local-inference topic, visit your repo's landing page and select "manage topics."