Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Refactor] br_rf_cafir #927

Merged
merged 6 commits into from
Jan 23, 2025
Merged

[Refactor] br_rf_cafir #927

merged 6 commits into from
Jan 23, 2025

Conversation

folhesgabriel
Copy link
Collaborator

@folhesgabriel folhesgabriel commented Jan 21, 2025 •
edited
Loading

Descrição do Pull Request

Este PR refatora a pipeline do conjunto br_rf_cafir

Detalhes técnicos

  1. Refatora a task antiga que selecionava arquivos para download no FTP em duas tasks
  • task_parse_api_metadata: extrai nomes dos arquivos e datas de atualização da API
  • task_decide_files_to_download: decide quais arquivos baixar, uma data específica ou a data mas recente
  1. refatora task de tratamento de dados
  • retira etapas de tratamento desnecessárias
  • tentei otimizar a velocidade de download dos dados requisições assíncronas. Porém, os resultados não foram bons.
  • Erros de time out do server ocorrem com frequência, mesmo com baixa quantidade de requisições paralelas (2-10). Mesmo utilizando um pool para reaproveitar a conexão os erros de time out ocorrem.
  • Para não complexificar a estrutura do crawler e dificultar o debug achei melhor manter o básico que funciona.

@folhesgabriel folhesgabriel self-assigned this Jan 21, 2025
@folhesgabriel folhesgabriel marked this pull request as draft January 21, 2025 16:00
@folhesgabriel folhesgabriel linked an issue Jan 21, 2025 that may be closed by this pull request
[fix] br_rf_cafir #915
Closed
@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 21, 2025 •
edited
Loading

Coverage

Tests Skipped Failures Errors Time
8 7 💤 0 ❌ 0 🔥 11.132s ⏱️

@folhesgabriel folhesgabriel requested a review from a team January 23, 2025 12:51
@folhesgabriel folhesgabriel marked this pull request as ready for review January 23, 2025 12:51
Winzen
Winzen approved these changes Jan 23, 2025
View reviewed changes
Copy link
Contributor

@Winzen Winzen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👌

@folhesgabriel folhesgabriel merged commit b8fb73f into main Jan 23, 2025
7 checks passed
@folhesgabriel folhesgabriel deleted the staging/refactor_cafir branch January 23, 2025 16:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Winzen Winzen Winzen approved these changes

Assignees

@folhesgabriel folhesgabriel

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[fix] br_rf_cafir
2 participants
@folhesgabriel @Winzen