Ingesting the Survey123 Data

For a full briefing of this feature, please visit the 19F handoff document.

---

When the admin user clicks “Upload Survey123 Data” on the password protected upload page, the system will request data from Survey123 for the selected state and forest. It then follows a two-part test to determine whether the data should be saved.

First, the system ignores any data that has not been marked as completed. It does this by seeing whether the “date completed” field in the survey is filled in.

Next, the system looks through each collection year’s object ID, a unique attribute which is automatically generated by Survey123. If our database has this object ID, then the system interprets this to mean that it already has this data, so it ignores this collection year. We do not check whether the data has changed nor do we make updates. This is why it is important for the future owner of this application to not ingest data for a specific state and forest until it is done. If the data has been saved and edits are required, an admin user should log into the mLab website and manually edit the data there.

If the database does not already have this object ID, then it considers this new data. It will save the data with the object ID, preventing future duplicate saving. It will also save various calculations relevant to the predictive model such as SPB and Clerids per two weeks.

It will also use the data to generate the name which our system uses to connect a year’s collection data to a specific forest. By convention, all state-level county forests are called the name of the forest followed by the state’s code, such as “Bankhead AL”. All national forests have two attributes, the name of the national forest and the ranger district name followed by “RD,” such as “Tombigbee RD.”

The system looks for a boolean variable from the Survey123 data called “isNF.” If this attribute is true, then the system will believe that the collection year data belongs to a national forest. It will look for forestName and rangerDistrictName attributes and save those, appending “RD” to the rangerDistrictName. If isNF is false, then it believes the relevant forest is a state forest. It will take the state and county attributes and combine them into the county/state name format.

Please note that this combined string will be the attribute used to match forests to each other. These names must be exactly and consistently the same in order to be considered the same forest by our system. Deviations will lead the app to believe that there are different forests.