Rewrite the app in Python Django 🚀

[raphodn]

I've been thinking lately...

Why ?

Having a couple of years of experience in Python Django, but only a few months now with Python FastAPI, I'm struggling to develop the backend as fast and solid as I wish..

Advantages of moving from FastAPI to Django

• a shell !
• ORM much better than sqlalchemy (joins, prefetch)
• API : use DRF : better serialization, better filtering (instead of https://github.com/arthurio/fastapi-filter)
• an admin interface 😍
• faster development, more solid backend, better manage the app's growth (my feeling is that FastAPI is good for prototyping/small projects, but harder to manage for bigger projects, with many dependencies (alembic, sqlalchemy, filter...) that are far from on-par with Django)

Disadvantages

• time to rewrite the app
• no typing
• API slower ? how slower ?
• cron managed by the server ? or django-q !

Extra info

• the initial POC - https://github.com/raphodn/open-prices-poc - was written in Django

[raphodn]

Todo

• django setup : feat(Django): init setup #362
• models : feat(Django): migrate models #364
• validation & tests : feat(Django): Model validation & tests #374 & refactor(Django): extra Price & Proof field validations #392
• api & serializers
  • Users : fea(Django): API Users endpoint #367
  • Location, Product & Price : feat(Django): API: Location, Product & Price GET endpoints #368
  • Proof : feat(Django): API: Proof endpoints #369
  • Location & Price POST : feat(Django): API: Location & Price POST endpoints #370
  • Price PATCH & DELETE : feat(Django): API: Price PATCH & DELETE endpoints #372
  • set Price.source & Proof.source : 3576ae0
  • update price_count (on create or delete) : feat(Django): update price_counts on Price create/delete #377
  • tests : feat(Django): API: Add & update tests #371
  • config, remove final slash, refactoring, fixes... : refactor(Django): improve API documentation #379 & refactor(Django): improve API serializers #381 & feat(Django): CORS configuration (allow all) #384 & fix(Django): last API fixes #385 & fix(Django): extra API fixes #389 & fix(Django): extra API fixes round 2 #401 & chore(tests): run Django tests instead of FastAPI #402 & fix(Django): improve some API returns #426
• authentication : feat(Django): authentication via API #365
• tasks, external APIs : feat(Django): fetch Product & Location data from external APIs (OFF, OSM) #373
  • on Price create, create/link Product & Location
  • on Proof create, create/link Location
  • on Product create, fetch from OFF
  • on Location create, fetch from OSM
• cron, async
  • daily product db sync : feat(Django): daily sync of products from OFF #376
  • daily data dump : feat(Django): daily DB backup task #380
  • make "fetch Product from OFF" & "fetch Location from OSM" async : refactor(Django): fetch data from OSM & OFF async #382
• devops (Docker, CI, deployment, Makefile) : chore(tests): run Django tests instead of FastAPI #402 & chore: fix CI #406 & chore: try to fix test CI #411 & chore: fix static configuration #415
• error mgmt with Sentry : chore: Add sentry config #400
• logging
• create init migrations files for each apps : feat(Django): add model init migrations #413
• performance (N+1) : perf(Django): Use select_related to avoid N+1 queries #403
• remove the fastapi code : refactor(Django): enlever le code fastapi #383
• update setup documentation : see Documentation need to be Updated #431
• speed optimizations : are index still valid ?

Questions

• Django Admin : generate a superuser ? (as we don't really manage users, nor store passwords)
• migrate lat & lon DecimalField to PointField ? PostGIS ?

[raphodn]

Which folder architecture ?

Existing

Repo	Config	Apps	api	web	deps	README	Choice
django tutorial	mysite/mysite	mysite/<apps>
MTES-MCT/envergo	envergo/config	envergo/envergo/<apps>			envergo/requirements.txt	envergo/README.md
MTES-MCT/aides-territoires	aides-territoires/src/core	aides-territoires/src/<apps>	inside each app	inside each app	aides-territoires/Pipfile	aides-territoires/README.md
quiz-anthropocene/admin-backend	admin-backend/core	admin-backend/<apps>	admin-backend/api		admin-backend/pyproject.toml	admin-backend/README.md
betagouv/Aidants_Connect	Aidants_Connect/aidants_connect	Aidants_Connect/<apps>	Aidants_Connect/api	Aidants_Connect/www	Aidants_Connect/pyproject.toml	Aidants_Connect/README.md
gip-inclusion/le-marche	le-marche/config	le-marche/lemarche/<apps>	le-marche/lemarche/api	le-marche/lemarche/www	le-marche/pyproject.toml	le-marche/README.md	looks good!
gip-inclusion/les-emplois	les-emplois/config	les-emplois/itou/<apps>	les-emplois/itou/api	les-emplois/itou/www	les-emplois/pyproject.toml	les-emplois/README.md	looks good!
betagouv/ma-cantine	ma-cantine/macantine	ma-cantine/data (models, ...)	ma-cantine/api	ma-cantine/web	ma-cantine/requirements.txt	ma-cantine/README.md
raphodn/open-prices-poc	open-prices-poc/config	open-prices-poc/<apps>	open-prices-poc/api	open-prices-poc/www	open-prices-poc/pyproject.toml	open-prices-poc/README.md

Todo

(closest to le-marche & les-emplois)

open-prices/config
open-prices/open_prices/<apps>
open-prices/open_prices/api
open-prices/open_prices/www
open-prices/pyproject.toml
open-prices/README.md

[raphodn]

Which tool to manage crons ?

Need

• cron : fetch & update all products from OFF every day
• async tasks : fetch new location details from OSM, new product details from OFF

Tools comparison

Name	Pros	Cons	Stars	Choice
django-crontab	- simple	- only manages crons - no admin integration	~800
django-celery	- Celery is famous	- cannot use DB as broker (since v4.2 ?) --> need to install Redis dependency	~1500
django-q	- manage tasks, cron... - can use DB as broker - admin integration	- not maintainted anymore !!	~1800
django-q2	- replaces django-q		~300	yes !

[raphodn]

@monsieurtanuki deployed in prod !

[raphodn]

created a discussion here with a quick todo of next/upcoming issues : #421