Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Add introduction page #431

Merged
merged 5 commits into from
Oct 17, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 3 additions & 32 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,38 +34,9 @@ We started logging incidents by server:

## Documentation

Link to [Github Page](https://openfoodfacts.github.io/openfoodfacts-infrastructure/)

The infrastructure documentation is as follows:

- [Overview](./docs/overview.md)

- [Mail](./docs/mail.md) - servers mail setup
- [Free Datacenter](./docs/free-datacenter.md) - Data center with main production servers
- [Linux Server](./docs/linux-server.md) - servers general setup
- [Mail](./docs/mail.md) - servers mail setup
- [An introduction to ZFS](./docs/zfs-overview.md) - ZFS is much used in our infrastructure
- [Proxmox](./docs/proxmox.md) - about proxmox management
- [CICD](./docs/cicd.md) - continuous integration and deployment
- [Observability](./docs/observability.md) - doc on monitoring / logs / etc.
- [Docker Onboarding](./docs/docker_onboarding.md)
- [Docker Infrastructure](./docs/docker_architecture.md)
- [Virtual Machines](#virtual-machines)

The main services:
- [MongoDB](./docs/mongodb.md) the MongoDB database
- [Redis](./docs/redis.md) we also use Redis
- [Open Food Facts Query](./docs/openfoodfacts-query.md) service computing aggregations

Some services:

- [Discourse](./docs/discourse.md) for forum
- [NGINX reverse proxy](./docs/nginx-reverse-proxy.md) the reverse proxy for OVH services
- [Folksonomy](./docs/folksonomy.md) user editable labels and values
- [Matomo](./docs/matomo.md) for web analytics
- [Producers sftp](./docs/producers_sftp.md) to push product updates on producer platform
- [Zammad](./docs/zammad.md) for support
- [Odoo](./docs/odoo.md) the CRM
See our [Introduction](./docs/introduction.md),
also on [Github Page](https://openfoodfacts.github.io/openfoodfacts-infrastructure/)


Also look at all install and post-mortem reports in [docs/reports](./docs/reports/)

Expand Down
133 changes: 133 additions & 0 deletions docs/introduction.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
# Introduction to Open Food Facts Infrastructure

Welcome to the Open Food Facts Infrastructure documentation!
This repository is dedicated to managing the infrastructure that powers Open Food Facts and its related projects. Our goal is to provide a reliable, scalable, and secure infrastructure to support the various services and applications that make up the Open Food Facts ecosystem.


## Main Practices

### Proxmox

Proxmox is an open-source server virtualization management solution that we use extensively in our infrastructure. It allows us to manage virtual machines (VMs) and containers (CTs) efficiently. Proxmox provides a web-based interface for easy management and monitoring of our virtualized environment.

Some software is installed / deployed in containers.
Docker deployments normally use a VM.

For more details about our Proxmox setup and management, see [Proxmox](./proxmox.md).

### Server Configuration Management

We manage server configurations using Git. Each server has a clone of this repository, and configuration files are symlinked to the appropriate locations. This allows us to track changes, maintain consistency, and easily roll back to previous configurations if needed. For more details, see [Explanation on server configuration with git](./explain-server-config-in-git.md).

### Continuous Integration and Continuous Delivery (CICD)

We use a lot CICD process to automate the integration of code changes. This ensures all tests pass and desired quality standards are met. Our CICD process includes automated testing, building Docker containers, and, for some software, deploying to pre-production and production environments. For more information, see [CICD](./cicd.md).

### Docker

Docker is a one of the key component of our infrastructure. We use Docker to containerize our applications, ensuring consistency and ease of deployment. Docker Compose is used for orchestration, allowing us to manage multi-container applications with ease. For more details,

see:
- [Docker at Open Food Facts](./docker.md).
- [Docker Onboarding](./docs/docker_onboarding.md)
- [Docker Infrastructure](./docs/docker_architecture.md)

### Observability

Observability allows us to monitor the health and performance of our systems, detect issues early, and gain insights into the behavior of our applications. We use a combination of tools and practices to achieve observability, including logging, metrics, and tracing. For more details, see [Observability](./observability.md).


### ZFS

We use a lot ZFS capabilities to store data on disk, and synchronize them accross servers thanks to Sanoid.

See:
- [ZFS Overview](./zfs-overview.md): An introduction to ZFS.
- [Sanoid](./sanoid.md): Information about using Sanoid for ZFS snapshots.


## Our Servers

Our infrastructure is hosted on multiple bare metal servers.
They are grouped in different data centers, usually forming a proxmox cluster.

See [Infrastructure Overview](./overview.md)

Some servers are graciously sponsored by [Fondation Free](https://www.fondation-free.fr/) (at [Scaleway](https://www.scaleway.com/)), [OVH](https://www.ovhcloud.com) and [Moji](https://moji.fr/)

For more details about our servers and their configurations, see the following pages:

- [Free Datacenter](./free-datacenter.md)
- [OVH Servers](./ovh-servers.md)
- [Moji Datacenter](./moji-datacenter.md)

## Production Architecture Overview

Our production architecture consists of different services to run Open Food Facts and sibling projects.
Those are deployed on different servers and different containers and virtual machines.

For a detailed overview of our production architecture, see [Production Architecture](./prod-architecture.md).

Other tools supporting the community are deployed in containers, some times on the same servers.

## Repository Structure

The repository is organized into several directories, each serving a specific purpose:

- `confs/`: Contains configuration files for various servers and services.
- `docker/`: Contains Docker-related files, including Docker Compose configurations.
- `docs/`: Contains documentation files, including this introduction.
- `docs/reports`: contains post mortem or log of installations.
- `scripts/`: Contains scripts for managing and maintaining the infrastructure.


## Services


### Important Services

- [Mail](./mail.md): Details about our mail setup.
- [NGINX reverse proxy](./nginx-reverse-proxy.md): The reverse proxy for all services


### Services Supporting the Main Open Food Facts Deployment

- [Product Opener](./product-opener.md): Backend that powers the Open Food Facts website and mobile apps.
- [Open Food Facts Query](./openfoodfacts-query.md): Service computing aggregations.
- [Postgres](./postgres.md): Information about our PostgreSQL setup and management.
- [MongoDB](./mongodb.md): Information about our MongoDB setup and management.
- [Redis](./redis.md): Details about our Redis setup and management.
- [Producers sftp](./producers_sftp.md): To push product updates on producer platform.
alexgarel marked this conversation as resolved.
Show resolved Hide resolved
- [Folksonomy](./folksonomy.md): User editable labels and values.

### Tools for the Community

- [Discourse](./discourse.md): For forum.
- [Matomo](./matomo.md): For web analytics.
- [Zammad](./zammad.md): For support.
- [Odoo](./odoo.md): The CRM.

## Additional Resources

Here are some additional resources that may be of interest:

- [Disks](./disks.md): Information about disk management and best practices.
- [How to mitigate crawlers on prod](./how-to-mitigate-crawlers-on-prod.md): Guide on mitigating crawlers on production.
- [How to resync ZFS replication](./how-to-resync-zfs-replication.md): Guide on resyncing ZFS replication.
- [Linux Server](./linux-server.md): General setup for Linux servers.
- [Rclone](./rclone.md): Information about using rclone.

### Incident logs

- [Logs off1](./logs-off1.md): Incident logs for off1 server.
- [Logs off2](./logs-off2.md): Incident logs for off2 server.
- [Logs off3](./logs-off3.md): Incident logs for off3 server.
- [Logs ovh1](./logs-ovh1.md): Incident logs for ovh1 server.
- [Logs ovh2](./logs-ovh2.md): Incident logs for ovh2 server.
- [Logs ovh3](./logs-ovh3.md): Incident logs for ovh3 server.

## You are welcome to contribute

We hope you find this documentation helpful and welcoming. If you have any questions or need further assistance, please feel free to reach out to us.

Happy contributing!
Loading