Skip to content

Commit

Permalink
Remove cloud providers
Browse files Browse the repository at this point in the history
  • Loading branch information
@jonaharagon
jonaharagon committed Nov 12, 2024
1 parent 851d30c commit 03ccfbf
Show file tree
Hide file tree
Showing 2 changed files with 19 additions and 122 deletions.
126 changes: 18 additions & 108 deletions docs/ai-chat.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,117 +16,39 @@ However, to improve the quality of LLMs, developers of AI software often use [Re

<details class="admonition info" markdown>
<summary>Ethical and Privacy Concerns about LLMs</summary>

AI models have been trained on massive amounts of public *and* private data. If you are concerned about these practices, you can either refuse to use AI or use [truly open-source models](https://proton.me/blog/how-to-build-privacy-first-ai), which publicly release their training datasets and therefore weren't trained on private data. One such model is [Olmoe](https://allenai.org/blog/olmoe) made by [Allenai](https://allenai.org/open-data).

[Ethical concerns](https://www.thelancet.com/journals/landig/article/PIIS2588-7500(24)00061-X/fulltext) about AI range from their impact on climate to their potential for discrimination.
</details>


## Cloud Providers

The AI chat cloud providers listed here do not train their models using your chats and do not retain your chats for more than a month, based on each service's privacy policy. However, there is **no guarantee** that these privacy policies are honored. Read our [full list of criteria](#criteria) for more information.

When using cloud-based AI chat tools, be mindful of the personal information you share. Even if a service doesn't store your conversations, there's still a risk of sensitive data being exposed or misused. To protect your privacy and security, **do not share sensitive information** related to health, finance, or other highly personal matters.

A quick **overview** of the two providers we recommend:

| Feature | DuckDuckGo AI | Brave Leo |
|---------|---------------|-----------|
| Tor Access | :material-check:{ .pg-green } Official onion service | :material-alert-outline:{ .pg-orange } Android-only (Orbot) |
| Rate Limits | :material-check:{ .pg-green } High | :material-alert-outline:{ .pg-orange } Low-Medium[^1] |
| Self-hosted Models | :material-close:{ .pg-red } | :material-check:{ .pg-green } |
| Web Search Integration | :material-close:{ .pg-red } | :material-check:{ .pg-green } |
| Multi-language Support | :material-check:{ .pg-green } | :material-alert-outline:{ .pg-orange } Limited |
| Account Required | :material-close:{ .pg-red } | :material-close:{ .pg-red } |
| Mobile Support | :material-check:{ .pg-green } | :material-check:{ .pg-green } only on Brave |

[^1]: Rate limits vary by model, with Llama having the lowest restrictions

### DuckDuckGo AI Chat

<div class="admonition recommendation" markdown>

![DuckDuckGo logo](assets/img/ai-chat/duckai.svg){align=right}

**DuckDuckGo AI Chat** is a web frontend for AI models. It is made by the popular [search engine provider](search-engines.md) of the same name.
It is available directly on [DuckDuckGo](https://duckduckgo.com), [duck.ai](https://duck.ai), or [DuckDuckGo onion site](https://duckduckgogg41xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/chat).

DuckDuckGo give you access to open-weights models from Meta and Mistral, as well as proprietary models from Anthropic and OpenAI. We strongly recommend you use open-weights models, because for those, no chat history is stored by Together.ai, the AI cloud platform DuckDuckGo uses to provide those models.
Furthermore, to protect your IP adress and prevent fingerprinting, DuckDuckGo proxies your chats through their servers.

[:octicons-home-16: Homepage](https://duck.ai){ .md-button .md-button--primary }
[:simple-torbrowser:](https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/chat){ .card-link title="Onion Service" }
[:octicons-eye-16:](https://duckduckgo.com/aichat/privacy-terms){ .card-link title="Privacy Policy" }
[:octicons-info-16:](https://help.duckduckgo.com){ .card-link title="Documentation" }

</div>

DuckDuckGo has agreements with their third-party providers that guarantee that they will not use your data for training their AI models. Proprietary model providers can keep a chat history for up to 30 days. For open-weights model, Duck uses the [together.ai](https://together.ai) AI cloud platform, and has disabled history for those chats.

<div class="admonition danger" markdown>
<p class="admonition-title">Proprietary Model Providers Retain Your Chats</p>
We advise against using proprietary models from Anthropic or OpenAI because those providers keep a chat history for up to 30 days.
</div>
<div class="admonition warning" markdown>
<p class="admonition-title">DuckDuckGo Doesn't Self-Host Open Models</p>
You will have to trust the together.ai cloud platform to honor their commitments to not store chats.
</div>

### Brave Leo

<div class="admonition recommendation" markdown>

![Brave Logo](assets/img/ai-chat/leo.svg){align=right}

**Brave Leo** is an AI assistant available inside the [Brave browser](https://brave.com), a browser we [recommend](tools/#private-web-browsers).

Brave Leo supports a variety of models, including open-weights models from Meta and Mistral, and proprietary models from Anthropic. We **strongly recommend** that you use **open-weights models**, because **Brave self-hosts them** and for those open-weights models, they **discards all chat data** after you close your session.

Additionally, the ["Bring Your Own Model"](https://brave.com/blog/byom-nightly/) (BYOM) feature allows you to use one of your local AI models directly in Brave.

[:octicons-home-16: Homepage](https://brave.com/leo){ .md-button .md-button--primary }
[:octicons-eye-16:](https://brave.com/privacy/browser/#brave-leo){ .card-link title="Privacy Policy" }
[:octicons-info-16:](https://github.com/brave/brave-browser/wiki/Brave-Leo){ .card-link title="Documentation" }

</div>
The default model is Mixtral, which has a low rate limit of 5 messages per hour. However, you can switch to the Llama model, which has "no" rate limits.

Leo can enhance its knowledge through web searches, similar to Microsoft Copilot. However, Brave's AI solution still faces challenges with multi-language support and contextual understanding.

<div class="admonition danger" markdown>
<p class="admonition-title">Page Content is Sent by Default</p>
By default, Brave Leo includes the webpage you are currently on as context for the AI model. While this can often be convenient, it also represents a privacy risk for pages with private information, such as your mailbox or social media. However, this feature cannot be globally disabled. Therefore, you'll need to **manually toggle off "Shape answers based on the page's contents"** for pages with PII.
</div>
<div class="admonition danger" markdown>
<p class="admonition-title">Proprietary Model Providers Retain Your Chats</p>
We advise against using Anthropic's Claude proprietary models because Anthropic keeps chat history for up to 30 days.
</div>

## Local AI Chat
</details>

**Running AI models locally** offers a more private and secure alternative to cloud-based solutions, as **your data never leaves your device** and is therefore never shared with third-party providers. This provides peace of mind and **allows you to share sensitive information**.

For the best experience, a dedicated GPU with sufficient VRAM or a modern system with fast LPDDR5X memory is recommended. Fortunately, it is possible to run smaller models locally even without a high-end computer or dedicated GPU. A computer with at least 8GB of RAM will be sufficient to run smaller models at lower speeds. Below is a table with more precise information :

<details class="admonition info" markdown>
<summary>Hardware Requirements for Local Models</summary>

Here are typical requirements for different model sizes:

- 7B parameter models: 8GB RAM minimum, 16GB recommended
- 13B parameter models: 16GB RAM minimum, 32GB recommended
- 70B parameter models: Dedicated GPU with 24GB+ VRAM recommended
- Quantized models (4-bit): Can run with roughly half these requirements
</details>

</details>

**To run AI locally, you need both an AI client and an AI model**.

### Download AI models
## Downloading AI models

There are many permissively licensed **models available to download**. **[Hugging Face](https://huggingface.co/models?library=gguf)** is a platform that lets you browse, research, and download models in common formats like GGUF. Companies that provide good open-weights models include big names like Mistral, Meta, Microsoft, and Google. But there are also many community models and 'fine-tunes' available. For consumer-grade hardware, it is generally recommended to use [quantized models](https://huggingface.co/docs/optimum/en/concept_guides/quantization) for the best balance between model quality and performance.
To help you choose a model that fit your needs, you can look at leaderboards and benchmarks. The most widely-used leaderboard is [LM Arena](https://lmarena.ai/), a "Community-driven Evaluation for Best AI chatbots". There is also the [OpenLLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard), which focus on the performance of open-weights models on common benchmarks like MMLU-PRO. However, there are also specialed benchmarks, that for example measure [emotional intelligence](https://eqbench.com/), ["uncensored general intelligence"](https://huggingface.co/spaces/DontPlanToEnd/UGI-Leaderboard), and many [others](https://www.nebuly.com/blog/llm-leaderboards).

<details class="admonition warning" markdown>
<summary>Model Security and Verification</summary>

When downloading AI models, especially from Hugging Face, it's important to verify their authenticity. Look for:

- Model cards with clear documentation
Expand All @@ -139,10 +61,10 @@ When downloading AI models, especially from Hugging Face, it's important to veri
3. Comparing this hash with the one you get after downloading (using tools like `sha256sum` on Linux/macOS or `certutil -hashfile file SHA256` on Windows)

Those steps help ensure you're not downloading potentially malicious models.
</details>

</details>

### AI chat clients
## AI chat clients

| Feature | [Kobold.cpp](#koboldcpp) | [Ollama](#ollama) | [Llamafile](#llamafile) |
|---------|------------|---------|-----------|
Expand All @@ -153,7 +75,7 @@ Those steps help ensure you're not downloading potentially malicious models.
| Custom Parameters | :material-check:{ .pg-green } | :material-close:{ .pg-red } | :material-alert-outline:{ .pg-orange } |
| Multi-platform | :material-check:{ .pg-green } | :material-check:{ .pg-green } | :material-alert-outline:{ .pg-orange } Size limitations on Windows |

#### Kobold.cpp
### Kobold.cpp

<div class="admonition recommendation" markdown>

Expand Down Expand Up @@ -186,7 +108,7 @@ Kobold shines best when you are looking for heavy customization and tweaking, su
Kobold.cpp might not run on computers without AVX/AVX2 support.
</div>

#### Ollama
### Ollama

<div class="admonition recommendation" markdown>

Expand All @@ -202,9 +124,11 @@ In addition to supporting a wide range of text models, Ollama also supports [LLa

<details class="downloads" markdown>
<summary>Downloads</summary>

- [:fontawesome-brands-windows: Windows](https://ollama.com/download/linux)
- [:simple-apple: macOS](https://ollama.com/download/mac)
- [:simple-linux: Linux](https://ollama.com/download/linux)

</details>

</div>
Expand All @@ -213,7 +137,7 @@ Ollama shines best when you're looking for an AI client that has great compatibi

It also simplifies the process of setting up a local AI chat, as it downloads the AI model you want to use automatically. For example, running `ollama run llama3.2` will automatically download and run the Llama 3.2 model. Furthermore, ollama maintains their own [model library](https://ollama.com/library/) where they host the files of various AI models. This ensures models are vetted for both performance and security, eliminating the need to manually verify model authenticity.

#### Llamafile
### Llamafile

<div class="admonition recommendation" markdown>

Expand All @@ -229,14 +153,18 @@ The Mozilla-run project also supports LLaVA. However, it does not support speech
[:octicons-lock-16:](https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file#security){ .card-link title="Security Policy" }
<details class="downloads" markdown>
<summary>Downloads</summary>

- [:fontawesome-solid-desktop: Desktop](https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file#quickstart)

</details>

</div>

<div class="admonition note" markdown>
<p class="admonition-title">Few Models Available</p>

Mozilla has only made llamafiles available for some Llama and Mistral models, while there are few third-party llamafiles available. Another issue is that Windows limits .exe files to 4GB, and most models are larger than that. To fix both of those issues, you can [load external weights](https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file#using-llamafile-with-external-weights).

</div>

## Criteria
Expand All @@ -245,16 +173,6 @@ Please note we are not affiliated with any of the projects we recommend. In addi

### Minimum Requirements

#### Cloud Providers

- The provider or third-parties they use must not use your chats for training.
- The provider or third-parties they use must not retain your chats for more than 30 days.
- Must be accessible privately (no account required, accepts requests from VPN users).
- Must provide models they host themselves or with a third-party that acts on their behalf.
- Must provide at least one model with high rate limits, to allow an user to use it for medium to heavy workloads.

#### Local AI clients

- Must be open-source.
- Must not send personal data, including chat data.
- Must be available on Linux.
Expand All @@ -266,14 +184,6 @@ Please note we are not affiliated with any of the projects we recommend. In addi

Our best-case criteria represent what we *would* like to see from the perfect project in this category. Our recommendations may not include any or all of this functionality, but those which do may rank higher than others on this page.

#### Cloud Providers

- Should not retain your chats.
- Should be accessible anonymously trough Tor.
- Should only offer self-hosted open-weights models.
- Should not be rate-limited.

#### Local AI clients
- Should be multi-platform.
- Should be easy to download and set up, such as having a one-click install process.
- Should have a built-in model downloader option.
Expand Down
15 changes: 1 addition & 14 deletions docs/tools.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -472,19 +472,6 @@ For encrypting your OS drive, we typically recommend using the encryption tool y

### AI Chat

#### Cloud Providers

<div class="grid cards" markdown>

- ![Duck AI logo](assets/img/ai-chat/duckai.svg){ .twemoji loading=lazy }[Duck AI](ai-chat.md#duckduckgo-ai-chat)
- ![Leo AI logo](assets/img/ai-chat/leo.svg){ .twemoji loading=lazy }[Brave Leo](ai-chat.md#brave-leo)

</div>

[Learn more :material-arrow-right-drop-circle:](ai-chat.md#cloud-providers)

#### Local AI

<div class="grid cards" markdown>

- ![Kobold logo](assets/img/ai-chat/kobold.png){ .twemoji loading=lazy }[Kobold.cpp](ai-chat.md#koboldcpp)
Expand All @@ -493,7 +480,7 @@ For encrypting your OS drive, we typically recommend using the encryption tool y

</div>

[Learn more :material-arrow-right-drop-circle:](ai-chat.md#local-ai-chat)
[Learn more :material-arrow-right-drop-circle:](ai-chat.md)

### Language Tools

Expand Down

0 comments on commit 03ccfbf

Please sign in to comment.