Perf: Pandas modin lib (#114)

* wip * perf: use modin instead of pandas * poetry lock * fix: merge
dataforgoodfr · Feb 26, 2024 · 510da04 · 510da04 · github-actions · Feb 26, 2024
1 parent 14a20d1
commit 510da04
Show file tree

Hide file tree

Showing 13 changed files with 2,641 additions and 276 deletions.
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -6,6 +6,7 @@ services:
       context: ./
       dockerfile: Dockerfile
     entrypoint: ["poetry", "run", "pytest", "--cov-report", "term:skip-covered", "--cov=quotaclimat", "--cov=postgres", "test/"]
+    #entrypoint: ["sleep", "1200"] # use to debug the container if needed
     environment:
       ENV: docker
       # CHANNEL: "fr3-idf"
@@ -16,6 +17,7 @@ services:
       POSTGRES_PASSWORD: password
       POSTGRES_HOST: postgres_db
       POSTGRES_PORT: 5432
+      MODIN_ENGINE: dask #TODO test me
     tty: true # colorize terminal
     volumes:
       - ./quotaclimat/:/app/quotaclimat/
@@ -142,6 +144,7 @@ services:
       MEDIATREE_PASSWORD:  /run/secrets/pwd_api
       MEDIATREE_AUTH_URL: https://keywords.mediatree.fr/api/auth/token/
       KEYWORDS_URL: https://keywords.mediatree.fr/api/subtitle/ # https://keywords.mediatree.fr/docs/#api-Subtitle-SubtitleList
+      MODIN_ENGINE: dask #TODO test me
     volumes:
       - ./quotaclimat/:/app/quotaclimat/
       - ./postgres/:/app/postgres/
File	Stmts	Miss	Cover	Missing
postgres
insert_data.py	46	7	85%	38–40, 59–61, 66
insert_existing_data_example.py	20	3	85%	25–27
postgres/schemas
models.py	71	15	79%	74–81, 91–92, 101–111
quotaclimat/data_analytics
analytics_signataire_charte.py	29	29	0%	1–67
bilan.py	108	108	0%	2–372
data_coverage.py	34	34	0%	1–94
exploration.py	125	125	0%	1–440
sitemap_analytics.py	118	118	0%	1–343
quotaclimat/data_ingestion
categorization_program_type.py	1	1	0%	1
config_youtube.py	1	1	0%	1
scaleway_db_backups.py	34	34	0%	1–74
scrap_chartejournalismeecologie_signataires.py	50	50	0%	1–169
scrap_sitemap.py	134	17	87%	27–28, 33–34, 66–71, 95–97, 138–140, 202, 223–228
scrap_tv_program.py	62	62	0%	1–149
scrap_youtube.py	114	114	0%	1–238
quotaclimat/data_ingestion/ingest_db
ingest_sitemap_in_db.py	59	41	31%	21–42, 45–65, 69–80
quotaclimat/data_ingestion/scrap_html
scrap_description_article.py	36	3	92%	19–20, 32
quotaclimat/data_processing/mediatree
api_import.py	177	102	42%	38–42, 47–50, 54–57, 63, 66–93, 99–114, 119–121, 146–153, 157–160, 164–170, 181–192, 195–199, 205, 231–232, 236, 240–259, 263–274
config.py	15	2	87%	7, 16
detect_keywords.py	108	3	97%	129–131
utils.py	64	21	67%	27–51, 54, 73–74
quotaclimat/data_processing/sitemap
sitemap_processing.py	41	27	34%	15–19, 23–25, 29–47, 51–58, 66–96, 101–103
quotaclimat/utils
channels.py	6	6	0%	1–95
climate_keywords.py	2	2	0%	3–35
healthcheck_config.py	29	14	52%	22–24, 27–38
logger.py	14	3	79%	22–24
plotly_theme.py	17	17	0%	1–56
sentry.py	10	2	80%	21–22
TOTAL	1575	961	39%