Data Preprocessing for HMIS Dataset

[balamurugan1603]

We've assigned people to work on preprocessing. You can post your findings or any abnormalities you found in the dataset structure over here.

[Dinesh-gan]

The HMIS dataset has two different formats

In All_India.csv

1. Before 2017-2018: 186 data points are recorded for every state.

2. From 2017-2018: 325 data points are recorded for every state.
   -- The Medical Description and Medical Code also change in this frame

3. The State Bifurcation of Andhra Pradesh Old adds the states of Telangana. Thus, from 2014-2015, 39 files including Telangana and Andhra Pradesh are present, before that 38 files with Andhra Pradesh Old were present.

4. M O Defence and M O Railways have been excluded from further processes since they are not states or UTs. They are also absent in 2019-2020 folder.

5. The 2017-2018/Telangana.csv has different column names and some additional texts, which have been removed, and the format across all states is standardized.

6.All the state files have been merged into two combined_csv_entire_india_before_2017.csv and combined_csv_entire_india_after_2017.csv files .
-- data has been organized for every "year, state, district"

7. All India .csv has also been organized similarly but for every "year, state".

8. Numerous District Reorganizations and District Name Changes have also occurred across many states in the given 2008-2020 time frame.