Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automate users scraping #124

Merged
merged 9 commits into from
Jul 10, 2024
Merged

Automate users scraping #124

Changes from 2 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
7957ac8
temporarily disable mymila source
nurbal May 27, 2024
999f8c6
cleaner "NotImplementedException" handling for `fetch_mymila` call
nurbal May 27, 2024
e342244
automate daily LDAP scraping
nurbal May 27, 2024
93d857f
fix tests
nurbal May 27, 2024
138c3ab
Add test for users versions count
nurbal Jun 3, 2024
a48625f
lint
nurbal Jun 3, 2024
0333861
update test
nurbal Jun 3, 2024
c157acf
ignore "record_start" and "record_end" fields
nurbal Jun 3, 2024
f65fbb0
fix test typo
nurbal Jun 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions sarc/ldap/acquire.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,11 +42,16 @@ def run(
cache_policy=cache_policy,
)

LD_users = fetch_mymila(
cfg,
LD_users,
cache_policy=cache_policy,
)
# MyMila scraping "NotImplementedError" is temporary ignored until we have a working fetching implementation,
# or a working workaround using CSV cache.
with using_trace(
"sarc.ldap.acquire", "fetch_mymila", exception_types=(NotImplementedError,)
Copy link
Collaborator Author

@nurbal nurbal Jun 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Je l'ai fait comme ça, de manière à gérer les deux cas:

  • en l'absence de fichier CSV de mymila (dans ce cas la lecture du cache échoue), fetch_mymila est appelée, une exception "Not Implemented" est lancée et gracieusement catchée, et MonMila est donc simplement ignoré
  • si on a le CSV, il est manuellement placé dans le répertoire de cache et directement lu.

Pour l'heure le traitement de ce fichier est l'objet d'une autre PR ( #120 ) mais pas encore Ok. Donc on ne place simplement pas le CSV en prod et le code fonctionne en l'état.

) as span:
LD_users = fetch_mymila(
cfg,
LD_users,
cache_policy=cache_policy,
)

# For each supervisor or co-supervisor, look for a mila_email_username
# matching the display name. If None has been found, the previous value remains
Expand Down
Loading