Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support to nested structs #1

Merged
merged 3 commits into from
Dec 2, 2024
Merged

feat: add support to nested structs #1

merged 3 commits into from
Dec 2, 2024

Conversation

raulb
Copy link
Member

@raulb raulb commented Nov 27, 2024

This will let us use things like this in conduit:

type RootFlags struct {
	// Global flags -----------------------------------------------------------

	// Conduit configuration file
	ConduitConfigPath string `long:"config.path" usage:"global conduit configuration file" persistent:"true" default:"./conduit.yaml"`

	// Version
	Version bool `long:"version" short:"v" usage:"show current Conduit version" persistent:"true"`

	conduit.Config
}

Where conduit.Config would be an updated version to what we currently have:

type Config struct {
	DB struct {
		// When Driver is specified it takes precedence over other DB related
		// fields.
		Driver database.DB

		Type   string `long:"db.type" usage:"database type; accepts badger,postgres,inmemory,sqlite"`
		Badger struct {
			Path string `long:"db.badger.path" usage:"path to badger DB"`
		}
		Postgres struct {
			ConnectionString string `long:"db.postgres.connection-string" usage:"postgres connection string, may be a database URL or in PostgreSQL keyword/value format"`
			Table            string `long:"db.postgres.table" usage:"postgres table in which to store data (will be created if it does not exist)"`
		}
		SQLite struct {
			Path  string `long:"db.sqlite.path" usage:"path to sqlite3 DB"`
			Table string `long:"db.sqlite.table" usage:"sqlite3 table in which to store data (will be created if it does not exist)"`
		}
	}

	API struct {
		Enabled bool `long:"api.enabled" usage:"enable HTTP and gRPC API"`
		HTTP    struct {
			Address string `long:"http.address" usage:"address for serving the HTTP API"`
		}
		GRPC struct {
			Address string `long:"grpc.address" usage:"address for serving the gRPC API"`
		}
	}

	Log struct {
		NewLogger func(level, format string) log.CtxLogger
		Level     string `long:"log.level" usage:"sets logging level; accepts debug, info, warn, error, trace"`
		Format    string `long:"log.format" usage:"sets the format of the logging; accepts json, cli"`
	}

	Connectors struct {
		Path string `long:"connectors.path" usage:"path to standalone connectors' directory"`
	}

	Processors struct {
		Path string `long:"processors.path" usage:"path to standalone processors' directory"`
	}

	Pipelines struct {
		Path           string
		ExitOnDegraded bool
		ErrorRecovery  struct {
			// MinDelay is the minimum delay before restart: Default: 1 second
			MinDelay time.Duration `long:"pipelines.error-recovery.min-delay" usage:"minimum delay before restart"`
			// MaxDelay is the maximum delay before restart: Default: 10 minutes
			MaxDelay time.Duration `long:"pipelines.error-recovery.max-delay" usage:"maximum delay before restart"`
			// BackoffFactor is the factor by which the delay is multiplied after each restart: Default: 2
			BackoffFactor int `long:"pipelines.error-recovery.backoff-factor" usage:"backoff factor applied to the last delay"`
			// MaxRetries is the maximum number of restarts before the pipeline is considered unhealthy: Default: -1 (infinite)
			MaxRetries int64 `long:"pipelines.error-recovery.max-retries" usage:"maximum number of retries"`
			// MaxRetriesWindow is the duration window in which the max retries are counted: Default: 5 minutes
			MaxRetriesWindow time.Duration `long:"pipelines.error-recovery.max-retries-window" usage:"amount of time running without any errors after which a pipeline is considered healthy"`
		}
	}

	ConnectorPlugins map[string]sdk.Connector

	SchemaRegistry struct {
		Type string `long:"schema-registry.type" usage:"schema registry type; accepts builtin,confluent"`

		Confluent struct {
			ConnectionString string `long:"schema-registry.confluent.connection-string" usage:"confluent schema registry connection string"`
		}
	}

	Preview struct {
		// PipelineArchV2 enables the new pipeline architecture.
		PipelineArchV2 bool `long:"preview.pipeline-arch-v2" usage:"enables experimental pipeline architecture v2 (note that the new architecture currently supports only 1 source and 1 destination per pipeline)"`
	}

	Dev struct {
		CPUProfile   string `long:"dev.cpuprofile" usage:"write CPU profile to file"`
		MemProfile   string `long:"dev.memprofile" usage:"write memory profile to file"`
		BlockProfile string `long:"dev.blockprofile" usage:"write block profile to file"`
	}
}

Notice that we'll be able to continue adding new configuration values, but only those with short or long will be taken into account. This will let us add automatically flags from this same config struct.

This is the current output using this in conduit:

go run ./cmd/conduit/main.go --help
Conduit CLI is a command-line that helps you interact with and manage Conduit.

Usage:
  conduit [flags]
  conduit [command]

Available Commands:
  help        Help about any command
  init        Initialize Conduit with a configuration file and directories.
  pipelines   Initialize and manage pipelines

Flags:
      --api.enabled                                            enable HTTP and gRPC API (default true)
      --config.path string                                     global conduit configuration file (default "/Users/rb/code/conduitio/conduit/conduit.yaml")
      --connectors.path string                                 path to standalone connectors' directory (default "/Users/rb/code/conduitio/conduit/connectors")
      --db.badger.path string                                  path to badger DB (default "/Users/rb/code/conduitio/conduit/conduit.db")
      --db.postgres.connection-string string                   postgres connection string, may be a database URL or in PostgreSQL keyword/value format
      --db.postgres.table string                               postgres table in which to store data (will be created if it does not exist) (default "conduit_kv_store")
      --db.sqlite.path string                                  path to sqlite3 DB (default "/Users/rb/code/conduitio/conduit/conduit.db")
      --db.sqlite.table string                                 sqlite3 table in which to store data (will be created if it does not exist) (default "conduit_kv_store")
      --db.type string                                         database type; accepts badger,postgres,inmemory,sqlite (default "badger")
      --dev.blockprofile string                                write block profile to file
      --dev.cpuprofile string                                  write CPU profile to file
      --dev.memprofile string                                  write memory profile to file
      --grpc.address string                                    address for serving the gRPC API (default ":8084")
  -h, --help                                                   help for conduit
      --http.address string                                    address for serving the HTTP API (default ":8080")
      --log.format string                                      sets the format of the logging; accepts json, cli (default "cli")
      --log.level string                                       sets logging level; accepts debug, info, warn, error, trace (default "info")
      --pipelines.error-recovery.backoff-factor int            backoff factor applied to the last delay (default 2)
      --pipelines.error-recovery.max-delay duration            maximum delay before restart (default 10m0s)
      --pipelines.error-recovery.max-retries int               maximum number of retries (default -1)
      --pipelines.error-recovery.max-retries-window duration   amount of time running without any errors after which a pipeline is considered healthy (default 5m0s)
      --pipelines.error-recovery.min-delay duration            minimum delay before restart (default 1s)
      --preview.pipeline-arch-v2                               enables experimental pipeline architecture v2 (note that the new architecture currently supports only 1 source and 1 destination per pipeline)
      --processors.path string                                 path to standalone processors' directory (default "/Users/rb/code/conduitio/conduit/processors")
      --schema-registry.confluent.connection-string string     confluent schema registry connection string
      --schema-registry.type string                            schema registry type; accepts builtin,confluent (default "builtin")
  -v, --version                                                show current Conduit version

Use "conduit [command] --help" for more information about a command.

flags.go Outdated Show resolved Hide resolved
@raulb raulb requested a review from hariso November 27, 2024 16:24
flags.go Show resolved Hide resolved
@raulb raulb requested a review from hariso December 2, 2024 11:55
@raulb raulb merged commit a81cef8 into main Dec 2, 2024
3 checks passed
@raulb raulb deleted the add-support-to-struct branch December 2, 2024 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants