PRSD-1023: Efficient LA Property Search

[isobel-softwire]

Ticket number

PRSD-1023

Goal of change

Makes LA property search efficient

Description of main change(s)

• Creates SQL script for generating load test data
• Adds copy of single line address to property ownership entity, alongside triggers for synchronisation
• Adds index on property ownership single line addresses
• Updates property search query accordingly

Anything you'd like to highlight to the reviewer?

• The SQL script generates 2,496,401 landlords and 4,992,802 property ownerships. This models the 2.82 million landlords and 4.7 million properties in the PRS, according to the English Private Landlord Survey
• Property searches take about 2 seconds on average, but this number varies based on how specific the search is. E.g. searching for a single letter will take longer.

Checklist

Delete any that are not applicable, and add explanation below for any that are applicable but haven't been done

• Test suite has been run in full locally and is passing
• Branch has been rebased onto main and run locally, with everything working as expected (both for your new feature
  and any related functionality)

[Travis-Softwire]

Does this approach not massively slow down the seeding of the DB with NGD address data?

[isobel-softwire]

This has been removed as we'll be referencing property ownerships from addresses rather than the other way around

[isobel-softwire]

Actually, querying the address table is much slower than querying the property ownership table, even with a partial index. I'll go back to referencing addresses from property ownerships. To combat slowing down NGD data loading, we can refresh the addresses once after loading finishes, rather than using this trigger.

[Travis-Softwire]

Intrigued that it takes longer - would expect it to be at least similar - but yes, if we can update them for each batch in one go during ingest instead of using the trigger that should at least limit the slow-down there

[Travis-Softwire]

Does the 2 second average drop if we disallow searches below a certain number of characters? Or is that the best case?

[Travis-Softwire]

I paused a bit over doing all of this in SQL though vs doing the manipulation on the host machine in either a Kotlin gradle task or a standalone node/TS script and just doing the batched inserts into the database.

[Travis-Softwire]

As mentioned on slack - on reflection this is fine given we'll use it so infrequently

[Travis-Softwire]

I've left some comments - I'm not 100% sure about doing this all in SQL as assuming that this is run from a dev laptop it'll all execute on the DB instance rather than the (much more powerful) host. It's also not the most readable language for non-trivial logic! But open to being persuaded that we should do it this way

[isobel-softwire]

Does the 2 second average drop if we disallow searches below a certain number of characters? Or is that the best case?

Not necessarily, as longer words that appear in many addresses (e.g. 'Manchester') will still return a large number of results.