OAK-11063: introduced inference service

[fabriziofortino]

Experimental support has been added for an external inference service designed to enrich ES indexes with dense vectors. The ElasticIndexDefinition now includes an optional Inference configuration, which consists of one or more properties and queries. The properties define how the external service should enrich the index, while queries specify when and how those queries should be executed. Additionally, the Inference configuration is stored within the Index Metadata, acting as a "contract" between OAK and the external service. Importantly, the service does not require direct access to OAK; instead, it simply reads the index metadata to determine how to enrich the index.

To ensure this feature functions correctly, the following tasks have been implemented:

• A :lastUpdated field has been added to each document. This allows the external inference service to avoid full index scans by identifying only those documents needing enrichment.
• The update method in ElasticIndexWriter has been revised. Previously, the /index API was used, which fully replaces a document, removing any additional fields enriched by the external service. To address this, the /update API is now used, as it preserves these enriched fields. However, the /update API is more resource-intensive than /index, as it must load an existing document instead of simply marking it as deleted. To maintain performance, the writer still uses the /index API during full reindexing or when an index is not externally mutable. In all other cases, it defaults to the /update API.

[nit0906]

Just a thought -

The time interval post which index would be scanned/updated by the external service is not being controlled by oak here - right ?

Could we do that ? Maybe even as part of the AsyncIndexerService - if it finds any diff to be indexed - it simply also pings the external service to add the vector enrichments ?

But if that would be too much frequent - maybe a separate service.

[steffenvan]

Just a comment from my side. Looks good. I like that we can keep the external API independent from Oak.

[fabriziofortino]

@nit0906 currently, the dependency between oak and the inference service happens at query time only.

Pinging the external service in the async indexer would create a dependency at index time too, that I would like to avoid.

The inference service support is marked as experimental for now. Let's see how it proceeds. If we realise something like is necessary, we can add it in the future.

[nfsantos]

Just a few minor comments.