Create methodology for matching to another entity

[eileenmcnaughton]

Currently when we use csvimporter we import one entity at a time. If we are importing a membership and wish to link to the contact we MUST provide contact_id or membership id.

However, if we are importing with the built in importer it is also an option to use external identifier

I think we should support any unique field on an entity as a proxy for that entity id

It's not currently possible to get a list of unique fields (or combinations) & I propose that we add an api action to do that - it might be Contact.getunique or similar (which would mean it would be implemented in the Generic class & trawl the DAO indices function). I considered it as a filter on getfields but there could be some unique combination indexes. We should agree what this might look like (@colemanw).

This would mean that for Contribution the unique fields 'trxn_id', 'invoice_id' would be picked up as metadata.

In terms of implementing into the csv import there are 2 options

1. add to this extension
2. add to https://github.com/saurabhbatra96/com.saurabhbatra96.civiimport by @saurabhbatra96

The latter is written in angular - the former is written in very very nasty php. (csvimport provides a thin layer over the core import classes but they are hard to work with).

My feeling is that @saurabhbatra96's extension would be easier to work with over the medium term, esp for devs that have angular experience. He is not currently maintaining it so probably agree with Saurabh whether to fork or get commit access to his repo if you go this way.

Known gaps - when working with @saurabhbatra96 extension I observed that it was not possible to download a list of fails etc as a csv. When I looked at how it is done in csvimport / any core imports I found that the latest csv is downloadable from a known url - this means it is possible to download someone else's import fails / matches etc. I think core should provide some sort of api to be able to have user-specific files (I would up using a drupal mechanism in my use case)

Additional contact matches
I think it also makes sense to be able to match by a chosen dedupe rule but there are some complexities there.... since they are not actually unique as the external_identifier etc are

[eileenmcnaughton]

Also am open to switching to api v4 if helpful

[saurabhbatra96]

I can help out by refactoring my code (I've been meaning to do that since forever) in case anybody wants to work on the extension.

[jamienovick]

ok, cool.

So it sounds as if there will be

1. A PR to core to add in the class as requested by @eileenmcnaughton above. I see API v4 has been suggested but I think stick with APIv3 for now and then we can add APIv4 support later if that is ok. @vinuvarshith from Compucorp to work on this.

2. Then a decision on which extension to support this in. @saurabhbatra96 I'll email you separately to provide a quote for a) code cleanup of your extension b) to provide the full feature comparative support for your angular extension. I can't guarantee we can fund this but I'll look at what costs we can cover. c) possibly looking at using method here for large imports:

https://github.com/civicrm/civicrm-core/blob/master/tools/extensions/org.civicrm.demoqueue/CRM/Demoqueue/Page/DemoQueue.php#L8-L40
more details: https://lists.civicrm.org/lists/arc/civicrm-dev/2018-04/msg00011.html
wiki: https://wiki.civicrm.org/confluence/display/CRMDOC/Howto+use+the+Queue+mechanism+in+your+extension

can you email me your contact details? @saurabhbatra96 and your availability to work on this over the next few weeks?

@eileenmcnaughton I'll make one point however - we find that the split between angular skills and PHP skills is quite ingrained and using an angular based extension will mean that finding devs who can work on both frontend and backend will be harder. We'll need to weigh this up.

[saurabhbatra96]

Hi @jamienovick , I'm absolutely jam-packed till the 12th of May (exams and a thesis submission 😢) but am willing to work on this after that.

• Re: the code cleanup - no need to pay me for that, it was long due anyway.
• Re: queuing for large imports - very interested to work on that.
• Re: full feature comparative support for the extension - not too excited about it but am willing to do it on a contractual basis (3 months initially and renewable)

A couple of disclaimers though, I've already committed to working on a GSoC project with Eileen during the summer so I'll be tied down with that and can only spare about 10-15 hours a week for this.
I'll be relocating in the last week of July and might/might not have time to spare (hence the 3 month contract and extended if I can spare some time after).

Email me here: saurabh.batra96 {at} gmail {dot} com