update!: Add local AI chat recommendations #2810
Conversation
Co-Authored-By: fria <[email protected]> Co-Authored-By: xe3 <[email protected]> Co-Authored-By: rollsicecream <[email protected]>
Co-authored-by: redoomed1 <[email protected]> Signed-off-by: Jonah Aragon <[email protected]>
✅ Your preview is ready!
|
theme/assets/img/ai-chat/ollama.svg
Outdated
There was a problem hiding this comment.
This isn't an actual SVG image, it appears to be a PNG embedded in an SVG file. This is not going to be well compatible with browsers (and is also just bad), so we need to replace it with the official https://ollama.com/public/ollama.png, or find a real vector file, or make our own (which I think @dngray has done before and knows how to do?)
(from #2525 (comment))
There was a problem hiding this comment.
Also we need a version of this that works in dark mode
There was a problem hiding this comment.
Isn't a PNG file is reasonable if there is no official SVG?
There was a problem hiding this comment.
Also we need a version of this that works in dark mode
Ollama doesn't have any dark mode logo.
There was a problem hiding this comment.
A PNG is fine, the problem is that this PNG is embedded in an SVG file which isn't fine. If we replace it with a regular PNG that could work.
We'll probably want to invert this logo and use it for dark mode, which should be fine as it's a simple image.
jonaharagon
force-pushed
the
ai-chat-local
branch
from
03ccfbf
to
b679d19
Compare
docs/ai-chat.md
Outdated
|
|
||
| </div> | ||
|
|
||
| Kobold shines best when you are looking for heavy customization and tweaking, such as for roleplaying purposes. It allows you to modify parameters such as the AI model temperature and the AI chat's system prompt. It also supports creating a network tunnel to access AI models from other devices, such as your phone. |
There was a problem hiding this comment.
| Kobold shines best when you are looking for heavy customization and tweaking, such as for roleplaying purposes. It allows you to modify parameters such as the AI model temperature and the AI chat's system prompt. It also supports creating a network tunnel to access AI models from other devices, such as your phone. | |
| Kobold is best if you are looking for heavy customization and tweaking. It allows you to modify parameters such as the AI model temperature and the AI chat's system prompt. It also supports creating a network tunnel to access AI models from other devices, such as your phone. |
Not sure if we need to mention roleplaying or other specific use-cases. Can we have an explanation of what temperature is?
There was a problem hiding this comment.
The temperature is defined in abreviations (it change the model creativity/randomness). We could explicitely define it, but it isn't recommended to change it, except for specifi use cases such as roleplaying (where you don't want your model to write a story about a princess every time...)
I think it is useful to mention use-cases to help user choose the ai client that best fit their needs.
Signed-off-by: Jonah Aragon <[email protected]>
|
This pull request has been mentioned on Privacy Guides Community. There might be relevant details there: https://discuss.privacyguides.net/t/add-ai-chat-tools/14347/175 |
Co-authored-by: Jonah Aragon <[email protected]> Signed-off-by: redoomed1 <[email protected]>
Co-authored-by: Jonah Aragon <[email protected]> Signed-off-by: redoomed1 <[email protected]>
Signed-off-by: redoomed1 <[email protected]>
docs/ai-chat.md
Outdated
| | Feature | [Kobold.cpp](#koboldcpp) | [Ollama](#ollama) | [Llamafile](#llamafile) | | ||
| |---------|------------|---------|-----------| | ||
| | GPU Support | :material-check:{ .pg-green } | :material-check:{ .pg-green } | :material-check:{ .pg-green } | | ||
| | Image Generation | :material-check:{ .pg-green } | :material-close:{ .pg-red } | :material-close:{ .pg-red } | | ||
| | Speech Recognition | :material-check:{ .pg-green } | :material-close:{ .pg-red } | :material-close:{ .pg-red } | | ||
| | Auto-download Models | :material-close:{ .pg-red } | :material-check:{ .pg-green } | :material-alert-outline:{ .pg-orange } Few models available | | ||
| | Custom Parameters | :material-check:{ .pg-green } | :material-close:{ .pg-red } | :material-alert-outline:{ .pg-orange } | | ||
| | Multi-platform | :material-check:{ .pg-green } | :material-check:{ .pg-green } | :material-alert-outline:{ .pg-orange } Size limitations on Windows | |
There was a problem hiding this comment.
| | Feature | [Kobold.cpp](#koboldcpp) | [Ollama](#ollama) | [Llamafile](#llamafile) | | |
| |---------|------------|---------|-----------| | |
| | GPU Support | :material-check:{ .pg-green } | :material-check:{ .pg-green } | :material-check:{ .pg-green } | | |
| | Image Generation | :material-check:{ .pg-green } | :material-close:{ .pg-red } | :material-close:{ .pg-red } | | |
| | Speech Recognition | :material-check:{ .pg-green } | :material-close:{ .pg-red } | :material-close:{ .pg-red } | | |
| | Auto-download Models | :material-close:{ .pg-red } | :material-check:{ .pg-green } | :material-alert-outline:{ .pg-orange } Few models available | | |
| | Custom Parameters | :material-check:{ .pg-green } | :material-close:{ .pg-red } | :material-alert-outline:{ .pg-orange } | | |
| | Multi-platform | :material-check:{ .pg-green } | :material-check:{ .pg-green } | :material-alert-outline:{ .pg-orange } Size limitations on Windows | | |
| | Local Client | GPU Support | Image Generation | Speech Recognition | Automatically Downloaded Models | Custom Parameters | | |
| |---|---|---|---|---|---| | |
| | [Kobold.cpp](#koboldcpp) | :material-check:{ .pg-green } | :material-check:{ .pg-green } | :material-check:{ .pg-green } | :material-close:{ .pg-red } | :material-check:{ .pg-green } | | |
| | [Ollama](#ollama) | :material-check:{ .pg-green } | :material-close:{ .pg-red } | :material-close:{ .pg-red } | :material-check:{ .pg-green } | :material-close:{ .pg-red } | | |
| | [Llamafile](#llamafile) | :material-check:{ .pg-green } | :material-close:{ .pg-red } | :material-close:{ .pg-red } | :material-alert-outline:{ .pg-orange } Few models available | :material-alert-outline:{ .pg-orange } | |
I removed the multi-platform comparison since that info is already available in multiple places (card descriptions and Downloads sections).
By the way, what is the distinction between the ❌ and the orange
There was a problem hiding this comment.
Honestly I just put it for Llamafile because I am not 100% sure you can modify parameters, I think you can from what I remember, but I am not sure.
|
@redoomed1 I did some new changes, but I am not authorised for any suggestions or push. So I include the file here. They add a new table for hardware requirements of AI models and rearrange some sections. |
|
Thanks @I-I-IT ! I'll go over your changes later and tag you as co-author in the commit. |
There was a problem hiding this comment.
All of these suggestions are from @I-I-IT's suggestions in the markdown file they provided here. I made some changes to improve readability and condense sections without stripping out any information.
Oops I entered the wrong username and email for co-authorship. Here's the commit for reference: 8ee05e8
You are welcome! I commented on a few questions/suggestions. Otherwise, this is looking pretty good. Splitting local and cloud was a good call, it allows for a much better structure. I was thinking that we could link to a video made by Naomi Brockwell about local AI Chat. They cover Ollama, and also how to choose an AI model. |
Co-authored-by: TechFanTheo <[email protected]> Signed-off-by: redoomed1 <[email protected]>
Signed-off-by: redoomed1 <[email protected]>
Signed-off-by: redoomed1 <[email protected]>
Signed-off-by: redoomed1 <[email protected]>
Signed-off-by: redoomed1 <[email protected]>
Signed-off-by: redoomed1 <[email protected]>
Signed-off-by: redoomed1 <[email protected]>
Signed-off-by: redoomed1 <[email protected]>
Signed-off-by: redoomed1 <[email protected]>
|
|
||
| ## AI Chat Clients | ||
|
|
||
| | Local Client | GPU Support | Image Generation | Speech Recognition | Automatically Downloaded Models | Custom Parameters | |
There was a problem hiding this comment.
The problem with this format is that it is mobile-unfriendly, as you have to scroll right to see all the info. Could we either revert it back, or change the format when we detect mobile screen size ?
|
|
||
| Kobold shines best when you are looking for heavy customization and tweaking, such as for roleplaying purposes. It allows you to modify parameters such as the AI model temperature and the AI chat's system prompt. It also supports creating a network tunnel to access AI models from other devices, such as your phone. | ||
|
|
||
| ### Ollama (CLI) |
There was a problem hiding this comment.
I think we could mention that ollama also has GUI community variants. https://github.com/ollama/ollama?tab=readme-ov-file#web--desktop
|
The page, without any admonitions is very long, perhaps too long. @jonaharagon for Ollama dark mode icon, the closest I could find is this one. (Notice it is a square with white surrounding the llama icon) |
|
They have an icon that would look good in dark and light mode on their GitHub README
|
Continuing work from: #2525 (comment)