Service to build a lookup of mappings between a list of strings and a list of objects

[josephjclark]

This is a super rough project outline. It has not been fully specced yet. It is mostly a thought experiment.

I think this is a very specific problem - by the time the solution is engineered, the problem will be fixed and it won't be needed!

Can we build it as an example? If we have good institutional knowledge about how to build services like this, maybe the next time the use case comes along we can build a microservice in 2 hours, rather than 2 weeks, which makes these ad-hoc solutions more appealing.

Requirement

Our services team has a requirement to take strings from commcare forms and convert them into FHIR object representations which map to particular codes.

There are a number of these mappings. One good example is mapping diagnoses from a picklist into a SNOMED code. There are like 250 diagnoses strings coming from commcare which need mapping to particular codes.

For example, the string "Dementia" needs to be mapped to an object like { system: 'smomed', code: '52448006', display: 'Dementia (disorder)' } (this is a semi made up example).

The mappings map strings to different standards, not just SNOMED and not just diagnoses

Usage

Here's how I see the service being used

• Give it a list of strings
• Give it some definition for the objects to map to (maybe literally an array of json objects, maybe some text reference from which we generate JSON)
• It will return a lookup table of strings:objects, like this:

{
   Dementia: {
      system: 'smomed',
      code: '52448006',
      display: 'Dementia (disorder)'
   }
}

• Copy and paste that table into a Job and, at runtime, map input data to the FHIR objects.

Design

We can ask a model eg chatgpt to do this for us. It will do an OK job of generating mappings, although those mappings need to be verified. And that's kinda hard

I picture a different design which might go like this:

• Give the service a list of input strings and list of definitions, like a JSON array of SNOMED codes
• The service will use natural language processing to find the closing matching definition
• If there's ambiguity it'll be flagged somehow, maybe the result contains an array of matching values
• It returns a lookup of mappings

Difficulty: Maybe the array of FHIR OMED codes doesn't exist, maybe that JSON needs to be generated. In which case it's a bit less like "pick the best match from the array" and more like "pick the best match and then generate the appropriate structure based on some input". Which is a bit harder.

Difficulty: even if the list of codes does exist as JSON, it's fiddly UX to upload the two lists to the AI service. Maybe we need to go and look for both the left and right datasets?

What I like about this design is that it's only doing one thing. AI is used to pick the best match (and maybe generate a structure), but its otherwise quite constrained. And we can design the output to be easily verifiable - each result object could have metadata explaining the mapping/

It's also quite broad? You just need a list of strings and a list of objects to map to. Maybe hints as to how the mapping works.

If you specialise the service to support a fix list of output/mapping formats, you can do more automation. Like maybe you give it a list of strings and specifically request to a) map them to SNOMED and b) map in the FHIR format. If the service has the capability to understand those formats, the user input because much much easier.

So perhaps this is a very application-specific layer on top of a more general lookup/mappings service.

[josephjclark]

Replaced by #109