Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge sitemap scaleway from polomarcus/barometre #83

Merged
merged 89 commits into from
Nov 24, 2023
Merged

Merge sitemap scaleway from polomarcus/barometre #83

merged 89 commits into from
Nov 24, 2023

Conversation

polomarcus
Copy link
Collaborator

@polomarcus polomarcus commented Oct 30, 2023 •
edited
Loading

To simplify the merge, i suggest a git push force

Features

  • add medias
  • parse some medias to get the article's description using BeautifulSoup

Tests

  • some tests with pytests and on the CI
  • Test coverage published on codecov

Ops

    • docker compose to run the project
    • Scaleway docker images - and serverless containers

To do

@polomarcus
docker: scrapper app
dbee944
@polomarcus
fix(docker): scrapper app name logs
e77da36
@polomarcus
docker: force container name
f996ac0
@polomarcus
wip: docker and config
47a8523
@polomarcus
wip: docker
ad666b0
@polomarcus
chores: update poetry lock
5268fed
@polomarcus
fix: docker start
e08c6e7
@polomarcus
fix: docker streamlit
5f2e593
@polomarcus
fix(streamlit): docker start
c282e67
@polomarcus
chores: poetry lock
a50d422
@polomarcus
chores: python 3.10 to 3.11
6e3b66e
@polomarcus
remove: scrap_sitemap
96d881a
@polomarcus
Merge pull request #1 from polomarcus/docker
efa6784 
Dockerize
@polomarcus
wip: add SQLAlchemy
d04c242
@polomarcus
wip(refacto): SQLAchemy + test
7755a26
@polomarcus
fix: ci
62db13c
@polomarcus
fix: ci install poetry
cad584d
@polomarcus
test(wip): save using pandas.to_sql
2949493
@polomarcus
ci: --dev-dependency
5a30d99
@polomarcus
chores: pytest update
862b9d9
@polomarcus
clean: notebooks should be in another repo
e2605a0
@polomarcus
lint
3648ef7
@polomarcus
ci: test job
e8fe510
@polomarcus
ci: need docker to run test
d2db6f0
@polomarcus
ci: directly use docker
e4899bb
@polomarcus
docker: wait for PG to be ready
913e522
@polomarcus
ci: docker exit code from container
a501277
@polomarcus
ci(test): use github action services to run PG
5e30fba
@polomarcus
ci(test): use poetry first
a1bb342
@polomarcus
ci(test): use github action services to run PG
e94be43
polomarcus and others added 21 commits October 25, 2023 14:30
@polomarcus
refacto: change PK due to publication date changing over time (#9)
1ae77b7 
Some medias changed their publication date instead of last modification,
so it messed up the PK


To do after deployment :
* delete duplicate due to new PK

```
psql
>
DELETE FROM
    sitemap_table a
        USING sitemap_table b
WHERE a.news_title = b.news_title;
```
[no ci]: bumping version
a5fa297
@polomarcus
fix(streamlit): use sqlachemy (#10)
f7c9db2 
Use env variable to connect to PG
[no ci]: bumping version
f0aa6af
@polomarcus
medias: add francebleu, nouvelobs, mediapart
ed429a6 
fix: nouvelobs
[no ci]: bumping version
b2efef2
@polomarcus
Feat: parse description meta tag for every news (#11)
32a7b8b 
To be the more generinic possible we parse this tag from every news :
 ```
<meta name="description" content="coucou">
```
https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta

Something to have in mind during first deployments : execution time and concurrency done with asyncio

:warning: websites made with JS cannot be parsed yet (need https://www.zenrows.com/blog/scraping-javascript-rendered-web-pages#requirements)
[no ci]: bumping version
ad7b97f
@polomarcus
feat: only save not known sitemap (PG already saved id) and sitemap l…
9c3c8e3 
…astmod/publication date

To avoid wasteful scrapping :
* we compare first the sitemaps we already know inside PG, then on the
difference of the sitemap.xml parsed, we continue to parse or not.
 * reading sitemap.xml we only keep 7 day-old news
[no ci]: bumping version
a1f9e11
@polomarcus
fix: corsematin
469a7dc
[no ci]: bumping version
c2828af
@polomarcus
refacto: healthcheck port renamed to PORT
e985894
@polomarcus
fix(description): log 20minutes missing hat
1f9a7f4
[no ci]: bumping version
b4d01b8
@polomarcus
log: log level + colors
8d070cb
[no ci]: bumping version
f87677e
@polomarcus
refacto: docker compose / CD push steps (#13)
d0b0f3d
[no ci]: bumping version
5faa5dd
@polomarcus
add medias: charentelibre, courrierpicard, ... (#14)
d296c1b
[no ci]: bumping version
b857083
@polomarcus polomarcus changed the title Merge sitemap scaleway WIP: Merge sitemap scaleway Oct 30, 2023
@polomarcus polomarcus force-pushed the merge-sitemap-scaleway branch from ecf0d86 to b857083 Compare October 30, 2023 17:04
@polomarcus polomarcus changed the title WIP: Merge sitemap scaleway Merge sitemap scaleway from polomarcus/barometre Oct 30, 2023
@estellerambier
Copy link
Collaborator

I take a look today 😬

estellerambier
estellerambier approved these changes Nov 24, 2023
View reviewed changes
Copy link
Collaborator

@estellerambier estellerambier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's do it! 🚀

Are you going to squash?

@polomarcus polomarcus merged commit 7e0ba98 into main Nov 24, 2023
1 check passed
@polomarcus polomarcus deleted the merge-sitemap-scaleway branch November 24, 2023 10:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@estellerambier estellerambier estellerambier approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@polomarcus @estellerambier