Write a guide about embedding Conduit

[simonl2002]

We should write a guide and example on how to embed Conduit in your own Go application. Here's part of a conversation with a user who played around with embedding Conduit, the guide should be more detailed.

---

You should start with conduit.NewRuntime, it will give you an object you can use to work with Conduit. Specifically the Orchestrator and ProvisioningService expose all important functionality. To get an idea of how to configure the runtime, you can have a look at the entrypoint used by the Conduit CLI. This should already be a good starting point. If you then want to provision a pipeline, you can either pass a configuration file to the provisioning service or use the orchestrator to manually create the pipeline, connectors and processors.

The database is used to store both things you mentioned, the pipeline configuration and its state. All Conduit needs is a key-value store. Out of the box we provide a way to store this data in a Postgres instance, locally in a badger DB or just in memory (but that gets wiped between restarts, so that's just useful for development purposes). You can also implement your own storage mechanism, it needs to implement the interface database.DB and pass the acceptance test, then you can provide it when configuring conduit as the DB driver.

[simonl2002]

Before we write a guide we should put some thinking about the "what" and "how" of embedding Conduit.

[simonl2002]

Relevant issue: #1274