feat: Add deployment config

[ayirr7]

Adds deployment configuration files which operate on a per-pipeline basis. Each file has an env section for general configuration (e.g. topics), a pipeline section for the local / dev configs, and allows for additional runtime-specific sections. For example, there is a flink section.

Adds a flag to the runner (-c) to denote container mode. This is useful in toggling between the local/dev configuration and container/non-local configuration. For the FlinkAdapter, this will mean that we can run the same example locally and in container mode down the road. Open to other suggestions for how to do this in the config formatting, etc.

[ayirr7]

I'd like us to move towards runtime-specific overrides (right now it's a full copy of the general pipeline config). I wanted to use Pydantic for some utils that make operations like deep updates of dictionaries easier, but I can't manage to install it.

[ayirr7]

Haven't split the PR yet because I think seeing whole diff is useful

[fpacifici]

A couple of high level suggestions:

• Define a schema for the config so we can validate it upfront rather than fail when a missing field is loaded. You can use jsonschema (yes. it is yaml but json is a subset so you can use jsonschema to validate).
• Let's have a config class that provides some solid abstracitons to the specific conmponents (i's recommend TypedDict that are lightweight but still type checkable).

[fpacifici]

?

[ayirr7]

I did this so I can test out parallelism of steps with Flink

[fpacifici]

ok

[fpacifici]

I think we are missing the concept of pipeline segments to define where we break chains.
Plus I am not sure the adapter specific config should be allowed to override any parameter. For example the flink version should not be able to override the kafka broker config. This makes me think that should be a separation between adapter specific config vs generic config.

What about something like this:

  graph TD;
     subgraph pipeline
       subgraph segment
          subgraph step
              common_config
              subgraph adapter_config
                  adapter1
                  adapter2
              end
           end
       end
    end

Loading

Where a pipeline is composed of semgnets (or chains) chains are wired up together in the same worker.
In each chain we provide config for each step (that needs config)
each step config can have a common config and a list of config elements per adapter.

There should be a way to provide some common environment like you did to map streams to topics.

[ayirr7]

Will it be the case that every Step will have access to the chain or segment it belongs to?

[fpacifici]

In #79 I abstracted away the type of source/sink. So we should pick the right source type in configuration. So you should have a representation of KafkaSource (with kafka paramters) and opening up the possibility to have other types.

[fpacifici]

Isn't parallelism a property of a segment that contains a sequence of steps.