Conduit CLI

[raulb]

Goal

Conduit Binary offers a variety of configuration options necessary for setting up your streaming service with Conduit. These meticulously designed flags are tailored to streamline the service's functionality:

$ conduit --help
Usage of conduit:
  -api.enabled
    	enable HTTP and gRPC API (default true)
  -config string
    	global config file (default "conduit.yaml")
  -connectors.path string
    	path to standalone connectors' directory (default "./connectors")
  -db.badger.path string
    	path to badger DB (default "conduit.db")
  -db.postgres.connection-string string
    	postgres connection string
  -db.postgres.table string
    	postgres table in which to store data (will be created if it does not exist) (default "conduit_kv_store")
  -db.type string
    	database type; accepts badger,postgres,inmemory (default "badger")
  -grpc.address string
    	address for serving the gRPC API (default ":8084")
  -http.address string
    	address for serving the HTTP API (default ":8080")
  -log.format string
    	sets the format of the logging; accepts json, cli (default "cli")
  -log.level string
    	sets logging level; accepts debug, info, warn, error, trace (default "info")
  -pipelines.exit-on-error
    	exit Conduit if a pipeline experiences an error while running
  -pipelines.path string
    	path to the directory that has the yaml pipeline configuration files, or a single pipeline configuration file (default "./pipelines")
  -processors.path string
    	path to standalone processors' directory (default "./processors")
  -version
    	prints current Conduit version

While these flags serve specific purposes for running the service, they assume an already familiarized user with steps to have a pipeline(s) successfully configured. This setting up process requires prior knowledge of the following elements:

1. Which available connectors Conduit offers (built-in or standalone) and how to configure them.
2. Which available processors can be used on Conduit (built-in or standalone) and how to configure them.
3. How to configure the desired pipeline to support customer needs.

What I’m proposing here is adding additional commands to extend a Conduit CLI to support those needs.

All commands, even when presenting potential interactivity, should be able to run via arguments and never require a prompt. For more information, see the CLIG guidelines on arguments and flags.

Assumptions

• Every command will have an alias to its singular/plural form. e.g.: connectors alias to connector. So they’ll be used interchangeably.
• Conduit CLI proposed here will live alongside Conduit’s service code. This does not take into account actions such as pipeline restart, pipeline stop. Once a pipeline is running, the way to interrupt the processing will be the same as up until now.
• If there’s only one pipeline registered, any --pipeline flag will be optional and it’ll use the only pipeline.

Step 0.

In order to accommodate new commands, I’d suggest moving existing root configuration options under start . To maintain backwards compatibility, these options will remain available on root, but will be hidden when typing conduit --help.

1. Create conduit start
2. Existing configuration options as previously shown will be moved under start.
3. This means that typing conduit --help will show only the commands added to root.
4. To explore further options based on each command, user will need to type conduit help COMMAND as it’s usual on CLIs.

Step 1. Getting started commands

When installing the binary for the first time, as a user, I’d like to have commands to help me getting started. Instructions we could refer to from our documentation for easy access.

Initializing a new pipeline: init

This command will initialize a simple pipeline that shows some basic features of Conduit. One example pipeline could be generator as a source and log as a destination (simplest setup, not requiring a provisioned resource).

The idea is that right after conduit init a user could simply do conduit run to show how Conduit works.

$ conduit init [NAME]

EXAMPLES
  $ conduit init
  $ conduit init my-first-pipeline

Additionally, this command could include a dialog in which we could present some example pipelines so anyone could use (postgres to log, etc...). This would require including some Docker images for easier installation.

Running the conduit service conduit start

# this is the equivalent to today's `conduit` default behavior.
$ conduit start [--pipelines.path] [...]

# previous flags will continue being available

If we wanted to introduce commands such as stop, or restart , this start command should return an ID that could be used to later be referenced. However, this discussion post is making the assumption that the CLI will run alongside existing Conduit binary.

Step 2. Extensibility

Installing a new pipeline pipelines install [--url] [--path]

This command will browse through the existing pipelines examples created in a specific repository for this purpose when --url is not provided. Specifying a pipeline that contains standalone connectors or processors will automatically add them.

When a url wasn’t specified, and optionally, to make the experience a bit less overwhelming, we could have a curated list on top and then others in case the user would like to explore more.

$ conduit pipeline install [--url] [--path]

EXAMPLES

	$ conduit pipeline install --url https://github.com/conduitio/pipeline-	examples/kafka-to-kafka.yml

	Installing the kafka-to-kafka pipeline to your Conduit instance... Done!
	Remember to edit it using `conduit pipeline edit kafka-to-kafka` to configure its connectors.

Edit a pipeline pipeline edit NAME [--path]

This command would open the $EDITOR to edit the associated pipeline configuration file.

$ conduit pipeline edit NAME [--path]

EXAMPLES
  $ conduit pipeline edit kafka-to-kafka 

Listing available pipelines pipelines ls [--path]

It’ll list the registered pipelines in --path (or /pipelines as default).

$ conduit pipeline ls [--path]

# This should indicate if a same file shows more than one pipeline
# The idea is to make it easier later on for removal if needed.

Removing a pipeline pipeline remove [--force]

This will require either deleting the yml file completely (in the event of this one having only one pipeline) or deleting just the pipeline that’s contained in that file.

$ conduit pipelines remove NAME

Describe a pipeline pipeline describe

Describes a pipeline showing name, connectors, and processors.

$ conduit pipelines describe NAME

EXAMPLES
  $ conduit pipeline describe kafka-to-kafka
  Source: kafka
     Processor: avro.encode
  Destination: kafka

Installing a new connector connector install [--url]

This command will install any standalone connectors available via ConduitIO Labs and will add it the connectors directory by default or to --path when specified via that flag. If --url is not provided, it’ll browse through a list of available connectors to be selected, and then installed.

$ conduit connectors install [--url] [--path]

$ conduit connectors install --url https://github.com/conduitio-labs/conduit-connector-mongo
Installing conduit-connector-mongo connector... Done!

Once installed, it’ll be available to use by any pipeline afterwards via connector add NAME --pipeline NAME.

Uninstall a connector connector uninstall NAME [--path]

This command will remove the specified standalone connector. If it’s used by a pipeline will require confirmation. The --confirm flag can be used to bypass the confirmation warning.

$ conduit connector uninstall NAME [--path] [--confirm]

$ conduit connector uninstall conduit-connector-mongo...
Uninstalling conduit-connector-mongo...

It's used by pipeline kafka-to-mongo. Are you sure you want to uninstall it? (y/n)?
> y
Uninstalling... Done!

Listing available connectors connectors ls

The ones that could be used locally (builtin + standalone). This list will indicate in what pipelines are being used (if they are). The idea of this command is to illustrate what connectors are already available if they were configured in a pipeline. This list would indicate how to reference the connector to make it easier.

$ conduit connectors ls

PLUGIN                               TYPE			PIPELINES
[email protected]	                     builtin			file-to-postgres
[email protected].        standalone		

Describe a connector: connectors describe NAME

This command could be used to describe the configuration needed, and if it’s used by a pipeline.

$ conduit connector describe [email protected]

NAME   DESCRIPTION                       REQUIRED  DEFAULT VALUE	EXAMPLE
url    HTTP URL to send requests to.     true		                https://...
...

Add a connector to a pipeline connectors add NAME --pipeline P-NAME --source | --destination

Once a connector has been installed, it’ll be ready to be used. If a user attempts to add a connector that’s not available yet, it’ll prompt to install it first. Adding will require configuration parameters that will depend on each connector. These options will be added to the pipeline, but the user will need to edit the pipeline afterwards. The optional options will be added to the pipeline configuration file, but commented out.

$ conduit connectors add plugin-name@version --pipeline pipeline --source

# e.g.:
$ conduit connectors add postgres --pipeline my-pipeline --destination 

Removing a connector connectors remove NAME --pipeline P-NAME

This will remove the connector configuration used by the specified pipeline. It’ll check if it’s used by a pipeline and will prompt configuration ahead of time.

$ conduit connectors remove NAME --pipeline PIPELINE-NAME 

$ conduit connector remove conduit-connector-mongo --pipeline my-pipeline...
Removing conduit-connector-mongo from pipeline my-pipeline...

It's used by pipeline kafka-to-mongo. Are you sure you want to remove it? (y/n)?
> y
Removing... Done!

Init a processor processor init [NAME] [--lang]

This will initialize a processor with the specified language (we’ll start with Go as default). Eventually, we could use the provided lang to indicate how to build one using a different language. We should have a processors template to have the same experience with pipelines and connectors. We’ll start with a simple processor such as the one in our documentation.

$ conduit processor init name [--lang]

EXAMPLES
	$ conduit processor init
	$ conduit processor init decode
	$ conduit processor init decode --lang go

Listing available processors processors ls

The ones that could be used locally (builtin + standalone ones):

$ conduit processors ls
NAME			TYPE
avro.decode 	builtin
avro.encode		builtin
base64.decode	builtin

Install a processor processors install [--url]

Similarly to connectors, when --url is not specified, it’ll browse through a list of available standalone processors.

$ conduit processor install [--url]

Uninstall a processor processors uninstall NAME [--force]

If a processor is being used by a pipeline it’ll mention it to confirm. Only standalone processors could be uninstalled.

$ conduit processor uninstall NAME [--force]
	
EXAMPLE

	$ conduit processor uninstall my-procesor
	The "my-processor" processor is currently used by pipeline kafka-to-kafka. To uninstall it, type the processor name.
  	> my-processor

Adding a new processor processors add NAME --pipeline P-NAME

Will add an available processor (all builtin + any standalone that could be available) to the specified pipeline.

$ conduit processors add NAME --pipeline P-NAME

Removing a processor processors remove NAME --pipeline P-NAME [--force]

Remove a processor used in a specified pipeline.

$ conduit processors remove NAME --pipeline P-NAME [--force]

Describe a processor processors describe PLUGIN

Show the available options for any processor that’s available (all builtin ones + the standalone that have been added).

$ conduit processor describe avro.decode

# auth.basic.password
type: string
description: This option is required if auth.basic.username contains a value. If both auth.basic.username and auth.basic.password are empty basic authentication is disabled.

# auth.basic.username
type: string
...

Building a processor processor build NAME [--path]

It depends on how Conduit was installed. We could try to package the tool alongside Conduit.

# GOARCH=wasm GOOS=wasip1 go build -o processor.wasm cmd/processor/main.go`
$ conduit processor build NAME [--path]

Step 3. Maintenance

Check pipelines via doctor

This will check whether there’s a more up to date version of conduit and if some connectors / processors could also be updated.

$ conduit doctor
# returns version, checks if there's a newer version of conduit, 
# plugin versions, etc

[alarbada]

Overall really useful commands @raulb ! The ones that just read from the setup are the most useful to me. I can always edit the pipeline configuration myself, but it is veeery useful to easily see what does conduit understand about the configuration that I've written into it, how many connectors are available, have I installed them correctly, etc.

[raulb]

Make sense, thanks for the review @alarbada ! Hopefully we can implement these soon 💪

[lovromazgon]

I agree with @alarbada that the most useful commands seem to be the ones for inspecting the Conduit instance, installing plugins, and utilities for developing a connector and processor. I'm not entirely convinced of those commands that add/remove a connector or processor to a pipeline config file, although I admit that some could be useful, since you then don't have to check what settings are available on a connector / processor. Also I first thought that removing a pipeline is as simple as removing the file so a command is redundant, then again, you have to know that removing the file will remove the pipeline, so a command would allow you to do that without knowing this piece of information. Maybe these commands have a purpose after all.

Additionally, I'm thinking that it would be nice to have something like config or similar, which tells me what DB is configured by default, what's the path to pipeline config files, to standalone connectors and to standalone processors. Essentially printing this config.

---

All in all, I think this design is already very good and it will be immensely valuable once we tackle this. Thanks @raulb 👏

[raulb]

@lovromazgon agreed! I thought about commands to managing different configurations which could be useful to be able run Conduit on different environments. I could also spike those out on this discussion.

[hariso]

Great stuff, @raulb I really like this! I agree with Lovro and Guillem, managing Conduit and helpers for developing connectors and processors sounds really useful! Here are some things that come to my mind:

1. What do you think about adding support for managing a remote Conduit instance?
2. We might also want to add commands for exporting and importing pipeline configs. It could be useful when a user wants to build and test a pipeline locally, and then import it into the Conduit Platform.

[raulb]

@hariso

1. What do you think about adding support for managing a remote Conduit instance?

I was purposely leaving that out of scope for now because it requires me doing some research on how that would work exactly, but I left this comment on conduit start:

If we wanted to introduce commands such as stop, or restart , this start command should return an ID that could be used to later be referenced. However, this discussion post is making the assumption that the CLI will run alongside existing Conduit binary.

2. We might also want to add commands for exporting and importing pipeline configs. It could be useful when a user wants to build and test a pipeline locally, and then import it into the Conduit Platform.

I also thought about this and felt this should be part of the Meroxa CLI. In my opinion, all utilities to bring Conduit to the platform should be separate from Conduit Itself. If anything, I would add all the platform utilities as a plugin, but not part of core Conduit CLI.

[nadilas]

Seems comprehensive with ease of use in mind. It would be a nice to have to mark the commands that are interactive and the ones that require no further input.

[raulb]

@nadilas I'm planning to use https://clig.dev/#interactivity as a guidance, so there should always be a way to programmatically use the CLI without requiring input.

[nadilas]

Nicely planned. I haven't even thought that far, I merely meant in the spec/docs and the cli help texts, but your plan is better.

[nickchomey]

I would also love to see this. It would make it easier to configure, deploy and interact with Conduit. Moreover, having cli commands available would make it easier for other tools (or even just bash scripts) to interact with Conduit via executing cli commands, rather than using the http api.

For example, I just learned of this tool, which might be useful to replace and augment the deprecated web UI https://github.com/dagu-org/dagu, and it appears to just run cli commands

@gedw99 do you have any thoughts on this?

[gedw99]

Hey @nickchomey

My own experience with cli systems is that a few things help to make it production / ops friendly.

piping. can you pipe it to easily build larger flows . Systems of systems .
Stdio is such a powerful thing.

config being multi sourced . Koanf does this allowing a config to be an amalgamation of many sources . This allows you to build reactive config.

Watch . Can the cli can be watched , you can build systems that self re-configure when the co fig changes.
This works well with koanf. Caddy is a good example of the watch without reloading manually concept.

ironically this way of approaching how a cli works is the same as why we use CDC systems. When the underlying data changes , the upstream / downstream systems can react.

you end up with unidirectional architectures with as few side effects as possible .

I hope that’s not too theoretical, but I only had a little bit of time to explain my thoughts .

those properties make it very easy to build a web gui on top of the cli . A simple htmx gui can be build where the server is the cli running in a watch mode .

Having the ability for all cli commands to output as json helps too . Then you can do request / response style piping , and so do a web GUI using htmx.