connman: wire up custom deserializer

docs/connection/new-connection-framework.md

@@ -0,0 +1,61 @@
+# New Connection Framework (WIP)
+
+## Usage
+
+Set environment variable `org.finos.legend.engine.enableNewConnectionFramework=true` when running `Server.java`
+
+## Goals and Objectives
+
+This is _an attempt_ to create cleanup the existing connection acquisition framework to make it more extensible as well as make it integrate better with work that has been done on identity/credential management by @epsstan as well as feedback from @pierredebelen. The core mission centers around: **"How do we make it easy for people (both platform developers and users) to acquire database connection given their identity?"**
+
+Fundamentally, the new connection framework makes DX for integrating new type of connection easier and consistent across different types of databases (not limited to only relational databases); in other words, adding support for `MongoDB` or `ElasticSearch` should require similar set of change to adding support for `Snowflake` or `BigQuery`, for example.
+
+The new connection framework API is also designed to be more declarative about the flows that it supports, hence, it's a continuation of the work that has been done on `DatabaseAuthenticationFlow` by @epsstan. In the new framework, when we define the connection manager/factory, we make it very explicit which flows are supported, we have have broken the flows down nicely into 2 functional components: `CredentialBuilder` - (can be chained) taking the identity and build a credential, `ConnectionBuilder` - taking a credential and acquire a connection; therefore these pieces are reusable in many flows, and we make it feasible for the system to support chained flows use case.
+
+Last but not least, we want to clean up the existing connection pooling code using `HikariCP` which has become fairly complex after many iterations.
+
+## Metamodel
+
+![framework-metamodel](./new-connection-framework-metamodel.png)
+
+
+
+## Connection Pooling
+
+## Migration Strategy
+ - For migration purpose, always assume to `credential vault`
+- [ ] Refactor further (breaking):
+  - [ ] Move GCP Applicaton Defaults Credential to the right module
+
+## Further Work / TODO
+
+- [ ] Deal with all the hacks @HACKY
+
+- [ ] Allow enabling the custom deseiralizer via configuration flag
+- [ ] Add protocol and grammar for the adapter type for `DatasourSpecification` and `AuthenticationStrategy`
+- [ ] Add pure processing roundtrip? - or can we be lazy about this?
+- [ ] Change `ConnectionManagerSelector` to no longer have the adapter code, just scan for the type of `RelationalDatabaseConnection.ds and authStrategy`, and have a converter with for `DatabaseType`, using ServiceLoader
+- [ ] Annotate Usage
+- [ ] Annotate Hacky stuffs
+- [ ] Reshape the converter to respect the triplets so we can make it easier to convert flow by flow
+- [ ] write docs about this:
+
+  - [ ] TODO: the reason we do lazy grammar/compiler/pure-metamodeling
+  - [ ] Plan in the future: code-generation
+  - [ ] Migration strategy, flow by flow
+  - [ ] Pooling work
+
+- [ ] hikari is asynch
+- [ ] easy to acquire the connection, hard to re-establish the connection when it's lost (esp for secure connect), we cannot assume we have the identity outside of the pool, --> let's do lazy, when we lose the connection, we will re-create it in the context of the next call, because we cannot cache the identity <-- CONFIRM it's not reconnecting on loss
+- [ ] `Ephrim` - check if Hikari has the ability to create connection synchronously
+  - [ ] Establishing connections is an expensive process
+  - [ ] What we do is we maintain connection pools to different database, think of it like a hashmap with values are stateful connection objects
+  - [ ] In some context, we want to do pushdown authentication, meaning, we cannot really share connection between people, An's connection to Snowflake_1 is different than Ephrim's connection to Snowflake_1, as such, the key of this hashmap is going to be made of the identity and the DB
+  - [ ] There are many mishaps that could happen to the connection we store in the pool, the connection is destroyed, the DB restarts, the connection expires, etc. --> we need to be able to re-establish the connection, Pierre said it create a new connection from scratch, it's not easy when we thought we found a connection in the pool and the connection has already lost, because the connection creation process of our pool (Hikari) is asynchronous, it means, connection is created on a different thread than the thread of the request, which contains the user's identity & credential, so as part of us trying to re-establish the connection, we need to somehow send the user's information from the request thread to the connection making thread, but we absolutely must not CACHE the user credential/identity anywhere because that's bad --> so how do we do this?
+  - [ ] 2 kerberos 1 thread
+- [ ] Refactor:
+  - [ ] Create subtype of `Connection`
+  - [ ] Realize the models into actual models, add Pure metamodel, etc.
+  - [ ] Test out global deserializer
+- [x] `Q - EPHRIM` Discuss about vault shortId, why don't we just use `VAULT` stuff? instead???? Why should we differentiate different types?
+

legend-engine-config/legend-engine-connection-integration-tests/src/test/java/org/finos/legend/engine/connection/test/TestSnowflakeConnection.java

@@ -33,7 +33,7 @@
 
 public class TestSnowflakeConnection
 {
-    public static class WithKeyPair extends AbstractConnectionFactoryTest<java.sql.Connection>
+    public static class ForSnowflakeWithKeyPairFlow extends AbstractConnectionFactoryTest<java.sql.Connection>
     {
         private static final String CONNECTION_INTEGRATION_TEST__SNOWFLAKE_PK = "CONNECTION_INTEGRATION_TEST__SNOWFLAKE_PK";
         private static final String CONNECTION_INTEGRATION_TEST__SNOWFLAKE_PK_PASSPHRASE = "CONNECTION_INTEGRATION_TEST__SNOWFLAKE_PK_PASSPHRASE";

legend-engine-config/legend-engine-server/src/main/java/org/finos/legend/engine/server/Server.java

@@ -48,7 +48,6 @@
 import org.finos.legend.connection.DatabaseSupport;
 import org.finos.legend.connection.LegendEnvironment;
 import org.finos.legend.connection.impl.CoreAuthenticationMechanismType;
-import org.finos.legend.connection.impl.HACKY__SnowflakeConnectionAdapter;
 import org.finos.legend.connection.impl.KerberosCredentialExtractor;
 import org.finos.legend.connection.impl.KeyPairCredentialBuilder;
 import org.finos.legend.connection.impl.RelationalDatabaseType;
@@ -80,7 +79,6 @@
 import org.finos.legend.engine.functionActivator.api.FunctionActivatorAPI;
 import org.finos.legend.engine.generation.artifact.api.ArtifactGenerationExtensionApi;
 import org.finos.legend.engine.language.hostedService.api.HostedServiceService;
-import org.finos.legend.engine.protocol.hostedService.deployment.HostedServiceDeploymentConfiguration;
 import org.finos.legend.engine.language.pure.compiler.api.Compile;
 import org.finos.legend.engine.language.pure.compiler.toPureGraph.PureModel;
 import org.finos.legend.engine.language.pure.grammar.api.grammarToJson.GrammarToJson;
@@ -95,7 +93,6 @@
 import org.finos.legend.engine.language.pure.modelManager.sdlc.SDLCLoader;
 import org.finos.legend.engine.language.pure.relational.api.relationalElement.RelationalElementAPI;
 import org.finos.legend.engine.language.snowflakeApp.api.SnowflakeAppService;
-import org.finos.legend.engine.protocol.snowflakeApp.deployment.SnowflakeAppDeploymentConfiguration;
 import org.finos.legend.engine.plan.execution.PlanExecutor;
 import org.finos.legend.engine.plan.execution.api.ExecutePlanLegacy;
 import org.finos.legend.engine.plan.execution.api.ExecutePlanStrategic;
@@ -120,10 +117,12 @@
 import org.finos.legend.engine.plan.execution.stores.service.plugin.ServiceStoreExecutorBuilder;
 import org.finos.legend.engine.plan.generation.extension.PlanGeneratorExtension;
 import org.finos.legend.engine.protocol.bigqueryFunction.metamodel.BigQueryFunctionDeploymentConfiguration;
+import org.finos.legend.engine.protocol.hostedService.deployment.HostedServiceDeploymentConfiguration;
 import org.finos.legend.engine.protocol.pure.v1.PureProtocolObjectMapperFactory;
 import org.finos.legend.engine.protocol.pure.v1.model.PureProtocol;
 import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.EncryptedPrivateKeyPairAuthenticationConfiguration;
 import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.UserPasswordAuthenticationConfiguration;
+import org.finos.legend.engine.protocol.snowflakeApp.deployment.SnowflakeAppDeploymentConfiguration;
 import org.finos.legend.engine.pure.code.core.PureCoreExtensionLoader;
 import org.finos.legend.engine.query.graphQL.api.debug.GraphQLDebug;
 import org.finos.legend.engine.query.graphQL.api.execute.GraphQLExecute;
@@ -291,9 +290,6 @@ public void run(T serverConfiguration, Environment environment)
             relationalExecution.setFlowProviderConfiguration(new LegendDefaultDatabaseAuthenticationFlowProviderConfiguration());
         }
         relationalExecution.setConnectionFactory(this.setupConnectionFactory(serverConfiguration.vaults));
-        relationalExecution.setRelationalDatabaseConnectionAdapters(Lists.mutable.of(
-                new HACKY__SnowflakeConnectionAdapter.WithKeyPair()
-        ));
 
         relationalStoreExecutor = (RelationalStoreExecutor) Relational.build(serverConfiguration.relationalexecution);
 

legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/ConnectionExtension.java

@@ -16,6 +16,7 @@
 
 import java.util.Collections;
 import java.util.List;
+import java.util.Map;
 
 public interface ConnectionExtension
 {
@@ -28,4 +29,9 @@ default List<AuthenticationMechanismType> getExtraAuthenticationMechanismTypes()
     {
         return Collections.emptyList();
     }
+
+    default Map<String, DatabaseType> getExtraDatabaseTypeMapping()
+    {
+        return Collections.emptyMap();
+    }
 }

legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/LegendEnvironment.java

@@ -78,15 +78,18 @@ protected LegendEnvironment(List<CredentialVault> vaults, List<DatabaseSupport>
         // load database types
         List<DatabaseType> databaseTypes = connectionExtensions.flatCollect(ConnectionExtension::getExtraDatabaseTypes);
         Map<String, DatabaseType> databaseTypesIndex = new LinkedHashMap<>();
+        Map<String, DatabaseType> mappedDatabaseTypesIndex = new LinkedHashMap<>();
+        connectionExtensions.forEach(extension -> mappedDatabaseTypesIndex.putAll(extension.getExtraDatabaseTypeMapping()));
         databaseTypes.forEach(databaseType ->
         {
             if (databaseTypesIndex.containsKey(databaseType.getIdentifier()))
             {
-                throw new RuntimeException(String.format("Found multiple authentication mechanisms with label '%s'", databaseType.getIdentifier()));
+                throw new RuntimeException(String.format("Found multiple database types with label '%s'", databaseType.getIdentifier()));
             }
             databaseTypesIndex.put(databaseType.getIdentifier(), databaseType);
         });
-        this.databaseTypesIndex = Maps.immutable.withAll(databaseTypesIndex);
+        mappedDatabaseTypesIndex.putAll(databaseTypesIndex);
+        this.databaseTypesIndex = Maps.immutable.withAll(mappedDatabaseTypesIndex);
 
         // load authentication configuration types
         Map<String, Class<? extends AuthenticationConfiguration>> authenticationConfigurationTypesIndex = new LinkedHashMap<>();

legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/RelationalConnectionExtension.java

@@ -15,11 +15,12 @@
 package org.finos.legend.connection.impl;
 
 import org.eclipse.collections.api.factory.Lists;
-import org.finos.legend.connection.AuthenticationMechanismType;
+import org.eclipse.collections.api.factory.Maps;
 import org.finos.legend.connection.ConnectionExtension;
 import org.finos.legend.connection.DatabaseType;
 
 import java.util.List;
+import java.util.Map;
 
 public class RelationalConnectionExtension implements ConnectionExtension
 {
@@ -28,4 +29,15 @@ public List<DatabaseType> getExtraDatabaseTypes()
     {
         return Lists.mutable.of(RelationalDatabaseType.values());
     }
+
+    @Override
+    public Map<String, DatabaseType> getExtraDatabaseTypeMapping()
+    {
+        return Maps.mutable.with(
+                org.finos.legend.engine.protocol.pure.v1.model.packageableElement.store.relational.connection.DatabaseType.H2.name(), RelationalDatabaseType.H2,
+                org.finos.legend.engine.protocol.pure.v1.model.packageableElement.store.relational.connection.DatabaseType.Snowflake.name(), RelationalDatabaseType.SNOWFLAKE,
+                org.finos.legend.engine.protocol.pure.v1.model.packageableElement.store.relational.connection.DatabaseType.Postgres.name(), RelationalDatabaseType.POSTGRES,
+                org.finos.legend.engine.protocol.pure.v1.model.packageableElement.store.relational.connection.DatabaseType.BigQuery.name(), RelationalDatabaseType.BIG_QUERY
+        );
+    }
 }