Existing frameworks that do JSON transformation

[csenn]

One key part of this project is transforming FHIR structures into explicitly defined output data structures that can be used to populate an SQL table. This is essentially a MAP procedure from FHIR -> SQL compatible data structure.

The input data could probably exist in various forms, such as FHIR JSON files, parquet files, FHIR Servers, blob stores, etc. However, for sake of this discussion, it could help to think of this procedure as a MAP from FHIR JSON -> FLAT JSON.

In this context, this is not a new problem, and in fact is a common need in ETL/ data integration pipelines. Therefore it would make considerable sense to attempt to gather some inspiration from other solutions.

For example, we have been looking at json based configuration that leverages FHIR path to pull select data out of FHIR objects. However, we have not discussed the practical issues such as how something like this could be run over 100 million patients. Many solutions out there are built around performance in mind. For instance, heavy use of data streaming (rather then pulling full objects into memory), solutions built on top of SPARK for distributed compute, etc.

Here a few libraries I have found that seem to be related, but I'm sure there are many more.

• https://github.com/bazaarvoice/jolt
• https://github.com/perpk/json-xform
• https://jsonpatch.com/
• https://www.jsoniq.org/
• https://github.com/logzio/sawmill

Googling JSON to JSON Transform also brings up plenty of links.

Even if a new solution is needed b/c this problem has different constraints (which we should be able to explicitly describe), there are probably many lessons to be learned from other approaches. And it's even possible that an "off the shelf" solution could adequately meed the needs here.

[niquola]

It would be interesting to investigate - could the FHIRPath based views be expressed in such tools

[csenn]

@niquola Just stumbled on an interesting project that may flatten FHIR JSON without any config, albeit into many tables. https://github.com/kindly/flatterer