129 export resource uuids #148

TraciebelWairimu · 2024-02-21T14:36:42Z

IMPORTANT: Where possible all PRs must be linked to a Github issue

Fixes #129

Engineer Checklist

I have run ./gradlew spotlessApply to check my code follows the project's style guide
I have built and run the efsity jar to verify my change fixes the issue and/or does not break the application

Wambere · 2024-02-22T09:39:38Z

importer/main.py

-@click.option("--csv_file", required=True)
+@click.option("--csv_file", required=False)
+@click.option("--export_resources", required=False)
+@click.option("--parameter", required=False)


We need to add default values for parameter, value and batch_size so that the export works even when these are not passed

Wambere · 2024-02-22T09:40:32Z

importer/main.py

+def write_csv(data, resource_type, fieldnames):
+    logging.info("Writing to csv file")
+    current_time = datetime.now().strftime("%Y-%m-%d-%H-%M")
+    csv_file = f"csv/{current_time}-export_{resource_type}.csv"


let's maybe create a folder inside /csv called exports just so this is cleaner and easier to find

Wambere · 2024-02-22T11:10:11Z

importer/main.py

+@click.option("--export_resources", required=False)
+@click.option("--parameter", required=False)
+@click.option("--value", required=False)
+@click.option("--batch_size", required=False)


this batch_size needs to be changed to limit

I label it as limit and not batch_size ?

Wambere · 2024-02-22T11:12:49Z

importer/main.py

@@ -17,6 +17,7 @@

 global_access_token = ""

+


remove this, and run black on these files

Wambere · 2024-02-22T11:17:08Z

importer/main.py

+            elements = ["name", "status", "id", "identifier", " organizations", "participants"]
+        else:
+            elements = []
+        for x in resources["entry"]:


we need to handle a case where the response returned has 0 resources and thus does not have "entry"

Wambere · 2024-02-22T11:21:50Z

importer/main.py

+        elif resource_type == "Organization":
+            elements = ["name", "active", "id", "identifier", "alias"]
+        elif resource_type == "CareTeam":
+            elements = ["name", "status", "id", "identifier", " organizations", "participants"]


It doesn't look like we are exporting the organizations and participants here, are we?

Wambere · 2024-02-22T11:33:45Z

importer/main.py

+
+# This function exports resources from the API to a csv file
+def export_resources_to_csv(resource_type, parameter, value, batch_size):
+    resource_type = get_valid_resource_type(resource_type)


let's change this (include in the doc ) to require the resource type exactly as it would be/work in a url so we don't have to do this. Might be a it confusing

Wambere · 2024-02-22T11:54:59Z

importer/main.py

@@ -974,5 +1052,6 @@ def main(
    total_time = end_time - start_time
    logging.info("Total time: " + str(total_time.total_seconds()) + " seconds")

+


When I black format the files the spaces are added back

Wambere · 2024-02-29T12:22:53Z

importer/README.md

+- `export_resources` can either be True or False, checks if it is True and exports the resources
+- The `parameter` is used as a filter for the resources. The set default parameter is "_lastUpdated", other examples include, "name"
+- The `value` is where you pass the actual parameter value to filter the resources. The set default value is "gt2023-01-01", other examples include, "Good Health Clinic 1"
+- The `limit` is the number of resources exported at a time


let's add the default here as well

Wambere · 2024-02-29T12:35:17Z

importer/main.py

+    if response[1] == 200:
+        resources = json.loads(response[0])
+        data = []
+        if "entry" in resources:


can we check this with resources["entry"] instead ?

Wambere · 2024-02-29T12:54:55Z

importer/main.py

+                    rl.append(value)
+                data.append(rl)
+            write_csv(data, resource_type, elements)
+            logging.info("Successfully written to csv")


There is nothing here checking that this was actually successful
I tried exporting with an invalid date time, python3 main.py --export_resources True --parameter _lastUpdated --value gt2023-02-29 --limit 20 --resource_type Location --log_level info
I got an error and no export but this still printed successful

Wambere · 2024-02-29T14:06:52Z

importer/test_main.py

+        self.test_fieldnames = ["name", "active", "method", "id", "identifier", "alias"]
+        write_csv(self.test_data, self.test_resource_type, self.test_fieldnames)
+        self.assertIsInstance(self.test_data, list)
+        self.assertEqual(len(self.test_data), 2)


Looks like these assert statements are not testing the result of your function call

Wambere · 2024-02-29T14:08:03Z

importer/test_main.py

+        self.assertEqual(len(self.test_data), 2)
+        current_time = datetime.now().strftime("%Y-%m-%d-%H-%M")
+        expected_csv_file_path = (
+            f"csv/exports/{current_time}-export_{self.test_resource_type}.csv"


It is possible that this current_time gotten in the test is different from the current_time gotten when the function was called

TraciebelWairimu force-pushed the 129-Export-Resource-uuids branch from 1ba56be to 610a82a Compare February 22, 2024 07:43

Wambere requested changes Feb 22, 2024

View reviewed changes

TraciebelWairimu force-pushed the 129-Export-Resource-uuids branch 2 times, most recently from 26596a8 to 74d6f73 Compare February 28, 2024 08:39

Wambere force-pushed the 129-Export-Resource-uuids branch 2 times, most recently from 74d6f73 to 9075ede Compare February 29, 2024 07:28

TraciebelWairimu and others added 7 commits February 29, 2024 12:06

Export resources from API endpoint tand write to csv file

4717601

Export resources test

c89178f

Fix test

18e042a

Update export_resources_function

a1dbfcc

Update export_resources_to_csv and documentation

61679a5

Fix test

6d1d3cc

Change 'Starting export...' placement

63fd41a

TraciebelWairimu force-pushed the 129-Export-Resource-uuids branch from 8a5daf1 to 63fd41a Compare February 29, 2024 09:06

Wambere requested changes Feb 29, 2024

View reviewed changes

Fix write_csv test, update requested changes

75a7bd7

pld previously approved these changes Mar 4, 2024

View reviewed changes

Update export location on docs

59dd5aa

Wambere dismissed pld’s stale review via 59dd5aa March 4, 2024 14:15

Wambere approved these changes Mar 4, 2024

View reviewed changes

Wambere merged commit f2e5290 into main Mar 4, 2024
4 checks passed

Wambere deleted the 129-Export-Resource-uuids branch March 4, 2024 14:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

129 export resource uuids #148

129 export resource uuids #148

TraciebelWairimu commented Feb 21, 2024

Wambere Feb 22, 2024

Wambere Feb 22, 2024

Wambere Feb 22, 2024 •

edited

Loading

TraciebelWairimu Feb 22, 2024

Wambere Feb 22, 2024

Wambere Feb 22, 2024

Wambere Feb 22, 2024 •

edited

Loading

Wambere Feb 22, 2024

Wambere Feb 22, 2024

TraciebelWairimu Feb 28, 2024

Wambere Feb 29, 2024

Wambere Feb 29, 2024

Wambere Feb 29, 2024

Wambere Feb 29, 2024

Wambere Feb 29, 2024

		@@ -974,5 +1052,6 @@ def main(
		total_time = end_time - start_time
		logging.info("Total time: " + str(total_time.total_seconds()) + " seconds")

129 export resource uuids #148

129 export resource uuids #148

Conversation

TraciebelWairimu commented Feb 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Wambere Feb 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Wambere Feb 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Wambere Feb 22, 2024 •

edited

Loading

Wambere Feb 22, 2024 •

edited

Loading