Skip to content

Commit

Permalink
Split DHIS2 exports into manageable units and retry failures (#5310)
Browse files Browse the repository at this point in the history
**Story card:**
[sc-11298](https://app.shortcut.com/simpledotorg/story/11298)

This PR aims to make the monthly exports of indicator data to
country-specific Dhis2 instances more robust by breaking them up into
retryable jobs that export data per facility identifier instead of all
at once.

It deletes the old exporter files and creates job files and their
corresponding specs. The jobs for each identifier are created in the
rake task now.
  • Loading branch information
qptr authored Nov 22, 2023
1 parent 2c712dd commit bb82bfa
Show file tree
Hide file tree
Showing 19 changed files with 440 additions and 645 deletions.
33 changes: 0 additions & 33 deletions app/exporters/bangladesh_dhis2_exporter.rb

This file was deleted.

55 changes: 0 additions & 55 deletions app/exporters/bangladesh_disaggregated_dhis2_exporter.rb

This file was deleted.

30 changes: 0 additions & 30 deletions app/exporters/dhis2/ethiopia_exporter.rb

This file was deleted.

68 changes: 0 additions & 68 deletions app/exporters/dhis2/helpers.rb

This file was deleted.

79 changes: 0 additions & 79 deletions app/exporters/dhis2_exporter.rb

This file was deleted.

49 changes: 49 additions & 0 deletions app/jobs/dhis2/bangladesh_disaggregated_exporter_job.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# frozen_string_literal: true

module Dhis2
class BangladeshDisaggregatedExporterJob < Dhis2ExporterJob
MIN_AGE = 15
MAX_AGE = 75
AGE_BUCKET_SIZE = 5

private

def facility_data_for_period(region, period)
{
htn_cumulative_assigned: PatientStates::Hypertension::CumulativeAssignedPatientsQuery.new(region, period).call,
htn_controlled: PatientStates::Hypertension::ControlledPatientsQuery.new(region, period).call,
htn_uncontrolled: PatientStates::Hypertension::UncontrolledPatientsQuery.new(region, period).call,
htn_missed_visits: PatientStates::Hypertension::MissedVisitsPatientsQuery.new(region, period).call,
htn_ltfu: PatientStates::Hypertension::LostToFollowUpPatientsQuery.new(region, period).call,
htn_dead: PatientStates::Hypertension::DeadPatientsQuery.new(region, period).call,
htn_cumulative_registrations: PatientStates::Hypertension::CumulativeRegistrationsQuery.new(region, period).call,
htn_monthly_registrations: PatientStates::Hypertension::MonthlyRegistrationsQuery.new(region, period).call,
htn_cumulative_assigned_adjusted: PatientStates::Hypertension::AdjustedAssignedPatientsQuery.new(region, period).call
}.transform_values { |patient_states| disaggregate_by_gender_age(patient_states, data_buckets(MIN_AGE, MAX_AGE, AGE_BUCKET_SIZE)) }
end

def data_elements_map
CountryConfig.dhis2_data_elements.fetch(:disaggregated_dhis2_data_elements)
end

def category_option_combo_ids
CountryConfig.dhis2_data_elements.fetch(:dhis2_category_option_combo)
end

def format_facility_period_data(facility_data, facility_identifier, period)
formatted_facility_data = []
facility_data.each do |data_element, values|
category_option_combo_ids.each do |combo, id|
formatted_facility_data << {
data_element: data_elements_map[data_element],
org_unit: facility_identifier.identifier,
category_option_combo: id,
period: reporting_period(period),
value: values.with_indifferent_access[combo] || 0
}
end
end
formatted_facility_data
end
end
end
23 changes: 23 additions & 0 deletions app/jobs/dhis2/bangladesh_exporter_job.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
module Dhis2
class BangladeshExporterJob < Dhis2ExporterJob
private

def facility_data_for_period(region, period)
{
htn_cumulative_assigned: PatientStates::Hypertension::CumulativeAssignedPatientsQuery.new(region, period).call.count,
htn_controlled: PatientStates::Hypertension::ControlledPatientsQuery.new(region, period).call.count,
htn_uncontrolled: PatientStates::Hypertension::UncontrolledPatientsQuery.new(region, period).call.count,
htn_missed_visits: PatientStates::Hypertension::MissedVisitsPatientsQuery.new(region, period).call.count,
htn_ltfu: PatientStates::Hypertension::LostToFollowUpPatientsQuery.new(region, period).call.count,
htn_dead: PatientStates::Hypertension::DeadPatientsQuery.new(region, period).call.count,
htn_cumulative_registrations: PatientStates::Hypertension::CumulativeRegistrationsQuery.new(region, period).call.count,
htn_monthly_registrations: PatientStates::Hypertension::MonthlyRegistrationsQuery.new(region, period).call.count,
htn_cumulative_assigned_adjusted: PatientStates::Hypertension::AdjustedAssignedPatientsQuery.new(region, period).call.count
}
end

def data_elements_map
CountryConfig.dhis2_data_elements.fetch(:dhis2_data_elements)
end
end
end
Loading

0 comments on commit bb82bfa

Please sign in to comment.