deploy: 93bf8f8

user-guide/sinks/kafka/index.html

@@ -898,17 +898,37 @@
     <ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
 
         <li class="md-nav__item">
-  <a href="#kafka-headers" class="md-nav__link">
-    Kafka Headers
+  <a href="#option-1-user-defined-kafka-sink" class="md-nav__link">
+    Option 1: User-Defined Kafka Sink
   </a>
 
 </li>
 
         <li class="md-nav__item">
+  <a href="#option-2-built-in-kafka-sink" class="md-nav__link">
+    Option 2: Built-in Kafka Sink
+  </a>
+
+    <nav class="md-nav" aria-label="Option 2: Built-in Kafka Sink">
+      <ul class="md-nav__list">
+
+          <li class="md-nav__item">
+  <a href="#kafka-headers" class="md-nav__link">
+    Kafka Headers
+  </a>
+
+</li>
+
+          <li class="md-nav__item">
   <a href="#example" class="md-nav__link">
     Example
   </a>
 
+</li>
+
+      </ul>
+    </nav>
+
 </li>
 
     </ul>
@@ -2321,17 +2341,37 @@
     <ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
 
         <li class="md-nav__item">
-  <a href="#kafka-headers" class="md-nav__link">
-    Kafka Headers
+  <a href="#option-1-user-defined-kafka-sink" class="md-nav__link">
+    Option 1: User-Defined Kafka Sink
   </a>
 
 </li>
 
         <li class="md-nav__item">
+  <a href="#option-2-built-in-kafka-sink" class="md-nav__link">
+    Option 2: Built-in Kafka Sink
+  </a>
+
+    <nav class="md-nav" aria-label="Option 2: Built-in Kafka Sink">
+      <ul class="md-nav__list">
+
+          <li class="md-nav__item">
+  <a href="#kafka-headers" class="md-nav__link">
+    Kafka Headers
+  </a>
+
+</li>
+
+          <li class="md-nav__item">
   <a href="#example" class="md-nav__link">
     Example
   </a>
 
+</li>
+
+      </ul>
+    </nav>
+
 </li>
 
     </ul>
@@ -2353,15 +2393,28 @@
 
 
 <h1 id="kafka-sink">Kafka Sink<a class="headerlink" href="#kafka-sink" title="Permanent link">&para;</a></h1>
+<p>Two methods are available for integrating Kafka topics into your Numaflow pipeline:
+using a user-defined Kafka Sink or opting for the built-in Kafka Sink provided by Numaflow.</p>
+<h2 id="option-1-user-defined-kafka-sink">Option 1: User-Defined Kafka Sink<a class="headerlink" href="#option-1-user-defined-kafka-sink" title="Permanent link">&para;</a></h2>
+<p>Developed and maintained by the Numaflow contributor community,
+the <a href="https://github.com/numaproj-contrib/kafka-java">Kafka Sink</a> provides a reliable and feature-complete solution for publishing messages to Kafka topics.</p>
+<p>Key Features:</p>
+<ul>
+<li><strong>Customization:</strong> Offers complete control over Kafka Sink configurations to tailor to specific requirements.</li>
+<li><strong>Kafka Java Client Utilization:</strong> Leverages the Kafka Java client for reliable message publishing to Kafka topics.</li>
+<li><strong>Schema Management:</strong> Integrates seamlessly with the Confluent Schema Registry to support schema validation and manage schema evolution effectively.</li>
+</ul>
+<p>More details on how to use the Kafka Sink can be found <a href="https://github.com/numaproj-contrib/kafka-java?tab=readme-ov-file#write-data-to-kafka">here</a>.</p>
+<h2 id="option-2-built-in-kafka-sink">Option 2: Built-in Kafka Sink<a class="headerlink" href="#option-2-built-in-kafka-sink" title="Permanent link">&para;</a></h2>
 <p>A <code>Kafka</code> sink is used to forward the messages to a Kafka topic. Kafka sink supports configuration overrides.</p>
-<h2 id="kafka-headers">Kafka Headers<a class="headerlink" href="#kafka-headers" title="Permanent link">&para;</a></h2>
+<h3 id="kafka-headers">Kafka Headers<a class="headerlink" href="#kafka-headers" title="Permanent link">&para;</a></h3>
 <p>We will insert <code>keys</code> into the Kafka header, but since <code>keys</code> is an array, we will add <code>keys</code> into the header in the
 following format.</p>
 <ul>
 <li><code>__keys_len</code> will have the number of <code>key</code> in the header. if <code>__keys_len</code> == <code>0</code>, means no <code>keys</code> are present.</li>
 <li><code>__keys_%d</code> will have the <code>key</code>, e.g., <code>__key_0</code> will be the first key, and so forth.</li>
 </ul>
-<h2 id="example">Example<a class="headerlink" href="#example" title="Permanent link">&para;</a></h2>
+<h3 id="example">Example<a class="headerlink" href="#example" title="Permanent link">&para;</a></h3>
 <div class="highlight"><pre><span></span><code><span class="nt">spec</span><span class="p">:</span>
 <span class="w">  </span><span class="nt">vertices</span><span class="p">:</span>
 <span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">kafka-output</span>

user-guide/sources/kafka/index.html

@@ -658,6 +658,20 @@
     </label>
     <ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
 
+        <li class="md-nav__item">
+  <a href="#option-1-user-defined-kafka-source" class="md-nav__link">
+    Option 1: User-Defined Kafka Source
+  </a>
+
+</li>
+
+        <li class="md-nav__item">
+  <a href="#option-2-built-in-kafka-source" class="md-nav__link">
+    Option 2: Built-in Kafka Source
+  </a>
+
+</li>
+
         <li class="md-nav__item">
   <a href="#faq" class="md-nav__link">
     FAQ
@@ -2326,6 +2340,20 @@
     </label>
     <ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
 
+        <li class="md-nav__item">
+  <a href="#option-1-user-defined-kafka-source" class="md-nav__link">
+    Option 1: User-Defined Kafka Source
+  </a>
+
+</li>
+
+        <li class="md-nav__item">
+  <a href="#option-2-built-in-kafka-source" class="md-nav__link">
+    Option 2: Built-in Kafka Source
+  </a>
+
+</li>
+
         <li class="md-nav__item">
   <a href="#faq" class="md-nav__link">
     FAQ
@@ -2365,7 +2393,21 @@
 
 
 <h1 id="kafka-source">Kafka Source<a class="headerlink" href="#kafka-source" title="Permanent link">&para;</a></h1>
-<p>A <code>Kafka</code> source is used to ingest the messages from a Kafka topic. Numaflow uses consumer-groups to manage offsets.</p>
+<p>Two methods are available for integrating Kafka topics into your Numaflow pipeline:
+using a user-defined Kafka Source or opting for the built-in Kafka Source provided by Numaflow.</p>
+<h2 id="option-1-user-defined-kafka-source">Option 1: User-Defined Kafka Source<a class="headerlink" href="#option-1-user-defined-kafka-source" title="Permanent link">&para;</a></h2>
+<p>Developed and maintained by the Numaflow contributor community,
+the <a href="https://github.com/numaproj-contrib/kafka-java">Kafka Source</a> offers a robust and feature-complete solution
+for integrating Kafka as a data source into your Numaflow pipeline.</p>
+<p>Key Features:</p>
+<ul>
+<li><strong>Flexibility:</strong> Allows full customization of Kafka Source configurations to suit specific needs.</li>
+<li><strong>Kafka Java Client Utilization:</strong> Leverages the Kafka Java client for robust message consumption from Kafka topics.</li>
+<li><strong>Schema Management:</strong> Integrates seamlessly with the Confluent Schema Registry to support schema validation and manage schema evolution effectively.</li>
+</ul>
+<p>More details on how to use the Kafka Source can be found <a href="https://github.com/numaproj-contrib/kafka-java/blob/main/README.md#read-data-from-kafka">here</a>.</p>
+<h2 id="option-2-built-in-kafka-source">Option 2: Built-in Kafka Source<a class="headerlink" href="#option-2-built-in-kafka-source" title="Permanent link">&para;</a></h2>
+<p>Numaflow provides a built-in <code>Kafka</code> source to ingest messages from a Kafka topic. The source uses consumer-groups to manage offsets.</p>
 <div class="highlight"><pre><span></span><code><span class="nt">spec</span><span class="p">:</span>
 <span class="w">  </span><span class="nt">vertices</span><span class="p">:</span>
 <span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">input</span>