Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add functionality to automatically select and use the most recent available period for all countries #4

Open
4 tasks
atowey-uchi opened this issue Dec 10, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@atowey-uchi
Copy link
Collaborator

atowey-uchi commented Dec 10, 2024

Description

Currently, users are required to manually input the period for data retrieval, which can lead to inconsistencies or errors if an outdated or unavailable period is selected. We need to enhance the functionality to automatically select and use the most recent available period for all countries to ensure data consistency and streamline the workflow.

The problem with this task is two-fold:

  1. Past and Future Dates with No Data:
    Some future dates (e.g., September 2025, 2026) appear in dropdowns but have no data because they are in the future. Simply selecting the most forward period (e.g., 2026) is incorrect as no data is available. Additionally, if a recent period (e.g., September 2025) has no data yet, pulling this period will result in an error, and we would need to default to an earlier period.

  2. Different Release Schedules for Countries:
    Not all countries have the same data release schedule. For example, some countries have data released every September, while Mozambique releases data in December for future years. This inconsistency means that an automated solution must be aware of these country-specific release schedules.

Currently, the output data includes a period column, which indicates which period the data corresponds to. We could consider adding a warning to notify users if the selected period is incorrect or missing data.

Proposed Tasks

  • Discuss preferences with users
  • Decide on the approach.

Option 1: Semi-automated Version: Use the Most Recent Yearly Period Based on Current Date for all Countries with Manual Updates

  • Example: If it's August 2025, use September 2024 data for all countries. If it is October 2025, use September 2025.
  • Challenge: This option may require additional warnings or error handling for missing data, and we would need to make country-by-country adjustments (e.g., Mozambique’s December release schedule). This solution will require ongoing maintenance to ensure it accommodates changes in release schedules.

Option 2: Automated Version: Run a Script to Scrape the Website and Find the Most Recent Data for Each Country

  • This would involve running a script before pull_naomi to check the website and identify the most recent available data for each country.
  • Challenge: If a country has data released for only one indicator during a specific period, we must decide whether to pull data for that period or find the most recent data across all indicators for that country. This would be a large undertaking.

Option 3: Continue Manual Input for Periods

  • Approach: Keep the current setup where users input the periods they want.
  • Consequence: No "most recent" parameter option, leaving the decision entirely to the user. This will not address the issue of inconsistencies and errors caused by incorrect period selection.

Option 4: Manual Update by Maintainer

  • Approach: The package maintainer manually updates the "most recent" parameter input whenever new data is released.
  • This would be a labor-intensive option, but it ensures the most recent data is always used.
  • Document decision, reasoning, and path forward
  • Implement any changes as necessary
@atowey-uchi atowey-uchi self-assigned this Dec 10, 2024
@atowey-uchi atowey-uchi added the enhancement New feature or request label Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant