Updating kafka schema and docs to provide better details (#885)

* Add better descriptions for kafka config in cdappconfig schema
* Update kafka provider doc
* Doc updates from 'make api-docs'
* Revert to NOT using enums for securityProtocol/saslMechanism fields

controllers/cloud.redhat.com/config/schema.json

@@ -201,16 +201,19 @@
             "description": "SASL Configuration for Kafka",
             "properties": {
                 "username": {
+                    "description": "Broker SASL username",
                     "type": "string"
                 },
                 "password": {
+                    "description": "Broker SASL password",
                     "type": "string"
                 },
                 "securityProtocol": {
-                    "description": "Deprecated: Use the top level securityProtocol field instead",
+                    "description": "Broker security protocol, expect one of either: SASL_SSL, SSL. DEPRECATED, use the top level securityProtocol field instead",
                     "type": "string"
                 },
                 "saslMechanism": {
+                    "description": "Broker SASL mechanism, expect: SCRAM-SHA-512",
                     "type": "string"
                 }
             },
@@ -222,22 +225,26 @@
             "description": "Broker Configuration",
             "properties": {
                 "hostname": {
+                    "description": "Hostname of kafka broker",
                     "type": "string"
                 },
                 "port": {
+                    "description": "Port of kafka broker",
                     "type": "integer"
                 },
                 "cacert": {
+                    "description": "CA certificate trust list for broker in PEM format. If absent, client should use OS default trust list",
                     "type": "string"
                 },
                 "authtype": {
                     "type": "string",
-                    "enum": ["mtls", "sasl"]
+                    "enum": ["sasl"]
                 },
                 "sasl": {
                     "$ref": "#/definitions/KafkaSASLConfig"
                 },
                 "securityProtocol": {
+                    "description": "Broker security procotol, expect one of either: SASL_SSL, SSL",
                     "type": "string"
                 }
             },

docs/antora/modules/providers/pages/kafka.adoc

@@ -46,26 +46,12 @@ CR. The CR will be placed in the environment specified in the `ClowdEnv`. The
 topic name will be modified as described above, to facilitate using the same
 Kafka instance for multiple apps in differing environments.
 
-ClowdEnv Config options available:
-
-- `clusterName`
-- `namespace`
-- `connectNamespace`
-- `connectClusterName`
-
 === app-interface
 
 In app-interface mode, the Clowder operator does not create any resources and
 simply passes through the topic names from the `ClowdApp` to the client
 config. The topics should be created via the usual app-interface means.
 
-ClowdEnv Config options available:
-
-- `clusterName`
-- `namespace`
-- `connectNamespace`
-- `connectClusterName`
-
 == Generated App Configuration
 
 The Kafka configuration appears in the cdappconfig.json with the following
@@ -78,67 +64,92 @@ the `name` attribute of a topic when connecting to Kafka.
 
 A helper is available below to facilitate quick access via a map.
 
-If a the `cacert` and `securityProtocol` fields are populated, then the kafka
-instance will be using TLS and the client should be configured as such
+== Authentication, SSL, and CA Certificates
+
+If the `securityProtocol` field is populated and its value is "SSL" or
+"SASL_SSL", then the client should use SSL to communicate with the broker.
 
 NOTE: The `securityProtocol` field has been deprecated from within the SASL
 stanza and in the future will only be available at the top level broker
 stanza.
 
-=== JSON structure
+If the `cacert` field is populated, clients should use this as the CA trust
+list when connecting to the broker. If it is not populated, then the broker's
+certificate is expected to be signed using a well-trusted certificate
+authority and clients should use the OS default CA trust list.
+
+=== Example JSON structures
 
+Broker is using SSL, a custom CA cert, and username/password auth:
 [source,json]
 ----
 {
   "kafka": {
-      "brokers": [
-          {
-              "hostname": "broker-host",
-              "port": 27015,
-              "securityProtocol": "SSL",
-              "cacert": "<some CA string>"
-          }
-      ],
-      "topics": [
-          {
-              "requestedName": "originalName",
-              "name": "someTopic",
-              "consumerGroupName": "someGroupName"
-          }
-      ]
+    "brokers": [
+      {
+        "authtype": "sasl",
+        "cacert": "<CA certs in PEM format>",
+        "hostname": "hostname.kafka.com",
+        "port": 9096,
+        "sasl": {
+          "password": "pw",
+          "saslMechanism": "SCRAM-SHA-512",
+          "securityProtocol": "SASL_SSL",
+          "username": "username"
+        },
+        "securityProtocol": "SASL_SSL"
+      }
+    ],
+    "topics": [
+      {
+        "requestedName": "originalName",
+        "name": "actualTopicName",
+      }
+    ]
   }
 }
 ----
 
-A User auth enabled Kafka will look like this
+Broker is unsecured (common in ephemeral environments)
 [source,json]
 ----
-{
   "kafka": {
-      "brokers": [
-          {
-              "hostname": "broker-host",
-              "port": 27015,
-              "authtype": "sasl",
-              "cacert": "-----BEGIN CERTIFICATE-----\nMIIDLTCCAhWgAwIBAgIJAPOWU.........",
-              "sasl":{
-                  "username": "kafkausername",
-                  "password": "kafkapassword",
-                  "securityProtocol": "SASL_SSL"
-              }
-          }
-      ],
-      "topics": [
-          {
-              "requestedName": "originalName",
-              "name": "someTopic",
-              "consumerGroupName": "someGroupName"
-          }
-      ]
+    "brokers": [
+      {
+        "hostname": "hostname.kafka.com",
+        "port": 9092
+      }
+    ],
+    "topics": [
+      {
+        "requestedName": "originalName",
+        "name": "actualTopicName",
+      }
+    ]
   }
 }
 ----
 
+Broker is using SSL but not SASL, and is signed by a commonly trusted CA (cacert not provided by Clowder)
+[source,json]
+----
+  "kafka": {
+    "brokers": [
+      {
+        "hostname": "hostname.kafka.com",
+        "port": 9093,
+        "securityProtocol": "SSL"
+      }
+    ],
+    "topics": [
+      {
+        "requestedName": "originalName",
+        "name": "actualTopicName",
+      }
+    ]
+  }
+}
+----
 
 === Client access
 
@@ -202,30 +213,28 @@ more information can be found in the API reference.
 :cyndi-operator: https://github.com/RedHatInsights/cyndi-operator#cyndi-operator
 :clowder-api-cyndi: https://redhatinsights.github.io/clowder/clowder/dev/api_reference.html#k8s-api-github-com-redhatinsights-clowder-apis-cloud-redhat-com-v1alpha1-cyndispec
 
-The *Cyndi* attribute of a ClowdApp definition is responsible for ensuring the Cyndi host
-syndication process is in place for the ClowdApp. On Clowder managed environments, the
-provider is also responsible of creating the CyndiPipeline resource, and configuring the underlying
-`Kafka Connect` so that the data syndication from Host Inventory database to the app's database
-works correctly.
-
 {kafka-connect}[Kafka Connect] is the core component used to perform Inventory’s host database
-syndication for some of the Insights platform applications, and uses an Operator to orchestrate
-the synchronization process.
-
-It does so by using a `CyndiPipeline` resource, which can be created by Clowder, and by injecting
-in the database secrets for both the target db (the application’s) and the host inventory db.
+syndication for some of the Insights platform applications, with the Cyndi operator managing the
+creation of KafkaConnector resources which facilitate the synchronization process via a `CyndiPipeline`
+resource.
 
 {kafka-connect}[Kafka Connect] streams table updates to keep data syndicated between the host
-inventory db and the application’s hosts view.
+inventory DB and the application database's "hosts" view.
+
+The *cyndi* attribute of a ClowdApp definition causes Clowder to ensure that a `CyndiPipeline` is in
+place for the ClowdApp. In ephemeral environments, Clowder will create the CyndiPipeline resource,
+itself, while in static environments like stage/prod, the CyndiPipeline is expected to already be
+present in the cluster. The ClowdEnvironment configuration controls Clowder's behavior in these
+different environments.
 
 Please refer to the corresponing projects for more information about the
 {project-cyndi}[Cyndi project], the `CyndiPipeline` resource or the {cyndi-operator}[Cyndi Operator]
 itself.
 
 === ClowdApp Configuration
 
-In order to request a `ClowdApp` to get the host syndication enabled by Cyndi, the `Cyndi` stanza
-needs to be used. A snippet of how that config would look like follows.
+In order to request a `ClowdApp` to get the host syndication enabled by Cyndi, the `cyndi` stanza
+needs to be used. A snippet of how that config would look like follows:
 
 [source,yaml]
 ----
@@ -243,19 +252,3 @@ The attributes are described in Clowder’s API Spec documentation {clowder-api-
 * *enabled* `[bool] default: true` - enables or disables the Cyndi dependency for this particular ClowdApp resource.
 * *appName* `[str] default: ''` - a string that sets the unique identifier of this ClowdApp on Cyndi.
 * *insightsOnly* `[bool] default: false` - enables the data syndication for all hosts or Insights hosts only.
-
-=== ClowdEnv Configuration
-
-The *Cyndi* provider will run in one of the two following modes, depending on whether if Clowder
-manages the environment or not;
-
-On non-Clowder managed environments (at the time of this writing, Stage and Production) Clowder
-will only check that the CyndiPipeline resource is available for the Clowdapp that is being
-reconciled in case it has the “cyndi” flag enabled in it’s template definition.
-Clowder will not try to create or update any resource related to Cyndi on environments it does not
-manage.
-
-On Clowder managed environments (Ephemeral environment) Clowder will configure and deploy Kafka
-using the Strimzi operator and will setup the CyndiPipeline to enable the host syndication process
-for the Clowdapps that require it on their spec files (see {clowder-api-cyndi}[Clowder API reference])
-

docs/appconfig/schema-definitions-brokerconfig-properties-authtype.md

@@ -21,5 +21,4 @@ https://cloud.redhat.com/schemas/clowder-appconfig#/definitions/BrokerConfig/pro
 
 | Value    | Explanation |
 | :------- | ----------- |
-| `"mtls"` |             |
 | `"sasl"` |             |

docs/appconfig/schema-definitions-brokerconfig-properties-cacert.md

@@ -4,7 +4,7 @@
 https://cloud.redhat.com/schemas/clowder-appconfig#/definitions/BrokerConfig/properties/cacert
 ```
 
-
+CA certificate trust list for broker in PEM format. If absent, client should use OS default trust list
 
 
 | Abstract            | Extensible | Status         | Identifiable            | Custom Properties | Additional Properties | Access Restrictions | Defined In                                                    |

docs/appconfig/schema-definitions-brokerconfig-properties-hostname.md

@@ -4,7 +4,7 @@
 https://cloud.redhat.com/schemas/clowder-appconfig#/definitions/BrokerConfig/properties/hostname
 ```
 
-
+Hostname of kafka broker
 
 
 | Abstract            | Extensible | Status         | Identifiable            | Custom Properties | Additional Properties | Access Restrictions | Defined In                                                    |

docs/appconfig/schema-definitions-brokerconfig-properties-port.md

@@ -4,7 +4,7 @@
 https://cloud.redhat.com/schemas/clowder-appconfig#/definitions/BrokerConfig/properties/port
 ```
 
-
+Port of kafka broker
 
 
 | Abstract            | Extensible | Status         | Identifiable            | Custom Properties | Additional Properties | Access Restrictions | Defined In                                                    |

docs/appconfig/schema-definitions-brokerconfig-properties-securityprotocol.md

@@ -4,7 +4,7 @@
 https://cloud.redhat.com/schemas/clowder-appconfig#/definitions/BrokerConfig/properties/securityProtocol
 ```
 
-
+Broker security procotol, expect one of either: SASL_SSL, SSL
 
 
 | Abstract            | Extensible | Status         | Identifiable            | Custom Properties | Additional Properties | Access Restrictions | Defined In                                                    |