[In progress] 0.8.0 (#168)

CHANGELOG.md

@@ -1,5 +1,32 @@
 # Karafka Web changelog
 
+## 0.8.0 (Unreleased)
+- **[Feature]** Provide ability to sort table data for part of the views (note: not all attributes can be sorted due to technical limitations of sub-components fetching from Kafka).
+- **[Feature]** Track and report pause timeouts via "Changes" view in Health.
+- **[Feature]** Introduce pending jobs visibility alongside of running jobs both in total and per process.
+- **[Feature]** Introduce states migrations for seamless upgrades.
+- **[Feature]** Introduce "Data transfers" chart with data received and data sent to the cluster.
+- **[Feature]** Introduce ability to download raw payloads.
+- **[Feature]** Introduce ability to download deserialized message payload as JSON.
+- [Enhancement] Split cluster info into two tabs, one for brokers and one for topics with partitions.
+- [Enhancement] Track pending jobs. Pending jobs are jobs that are not yet scheduled for execution by advanced schedulers.
+- [Enhancement] Rename "Enqueued" to "Pending" to support jobs that are not yet enqueued but within a scheduler.
+- [Enhancement] Make sure only running jobs are displayed in running jobs
+- [Enhancement] Improve jobs related breadcrumbs
+- [Enhancement] Display errors backtraces in OSS.
+- [Enhancement] Display concurrency graph in OSS.
+- [Enhancement] Support time ranges for graphs in OSS.
+- [Enhancement] Report last poll time for each subscription group.
+- [Enhancement] Show last poll time per consumer instance.
+- [Enhancement] Display number of jobs in a particular process jobs view.
+- [Enhancement] Promote "Batches" chart to OSS.
+- [Enhancement] Promote "Utilization" chart to OSS.
+- [Fix] Fix times precisions that could be incorrectly reported by 1 second in few places.
+- [Fix] Fix random order in Consumers groups Health view.
+- [Change] Rename "Busy" to "Running" to align with "Running Jobs".
+- [Change] Rename "Active subscriptions" to "Subscriptions" as process subscriptions are always active.
+- [Maintenance] Introduce granular subscription group contracts.
+
 ## 0.7.10 (2023-10-31)
 - [Fix] Max LSO chart does not work as expected (#201)
 
@@ -29,6 +56,15 @@
 - [Fix] Cache assets for 1 year instead of 7 days.
 - [Fix] Remove source maps pointing to non-existing locations.
 - [Maintenance] Include license and copyrights notice for `timeago.js` that was missing in the JS min file. 
+- [Refactor] Rename `ui.show_internal_topics` to `ui.visibility.internal_topics_display`
+
+### Upgrade Notes
+
+**NO** rolling upgrade needed. Just configuration update.
+
+1. If you are using `ui.visibility_filter` this option is now `ui.visibility.filter` (yes, only `.` difference).
+2. If you are using a custom visibility filter, it requires now two extra methods: `#download?` and `#export?`. The default visibility filter allows both actions unless message is encrypted.
+3. `ui.show_internal_topics` config option has been moved and renamed to `ui.visibility.internal_topics`.
 
 ## 0.7.4 (2023-09-19)
 - [Improvement] Skip aggregations on older schemas during upgrades. This only skips process-reports (that are going to be rolled) on the 5s window in case of an upgrade that should not be a rolling one anyhow. This simplifies the operations and minimizes the risk on breaking upgrades.

Gemfile.lock

@@ -1,9 +1,9 @@
 PATH
   remote: .
   specs:
-    karafka-web (0.7.10)
+    karafka-web (0.8.0)
       erubi (~> 1.4)
-      karafka (>= 2.2.9, < 3.0.0)
+      karafka (>= 2.2.14, < 3.0.0)
       karafka-core (>= 2.2.4, < 3.0.0)
       roda (~> 3.68, >= 3.69)
       tilt (~> 2.0)
@@ -36,14 +36,14 @@ GEM
     ffi (1.16.3)
     i18n (1.14.1)
       concurrent-ruby (~> 1.0)
-    karafka (2.2.9)
-      karafka-core (>= 2.2.2, < 2.3.0)
-      waterdrop (>= 2.6.10, < 3.0.0)
+    karafka (2.2.14)
+      karafka-core (>= 2.2.7, < 2.3.0)
+      waterdrop (>= 2.6.11, < 3.0.0)
       zeitwerk (~> 2.3)
-    karafka-core (2.2.4)
+    karafka-core (2.2.7)
       concurrent-ruby (>= 1.1)
-      karafka-rdkafka (>= 0.13.6, < 0.14.0)
-    karafka-rdkafka (0.13.6)
+      karafka-rdkafka (>= 0.13.9, < 0.15.0)
+    karafka-rdkafka (0.14.0)
       ffi (~> 1.15)
       mini_portile2 (~> 2.6)
       rake (> 12)
@@ -82,7 +82,7 @@ GEM
     tilt (2.3.0)
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
-    waterdrop (2.6.10)
+    waterdrop (2.6.11)
       karafka-core (>= 2.2.3, < 3.0.0)
       zeitwerk (~> 2.3)
     webrick (1.8.1)

karafka-web.gemspec

@@ -17,7 +17,7 @@ Gem::Specification.new do |spec|
   spec.licenses    = %w[LGPL-3.0 Commercial]
 
   spec.add_dependency 'erubi', '~> 1.4'
-  spec.add_dependency 'karafka', '>= 2.2.9', '< 3.0.0'
+  spec.add_dependency 'karafka', '>= 2.2.14', '< 3.0.0'
   spec.add_dependency 'karafka-core', '>= 2.2.4', '< 3.0.0'
   spec.add_dependency 'roda', '~> 3.68', '>= 3.69'
   spec.add_dependency 'tilt', '~> 2.0'

lib/karafka/web.rb

@@ -53,7 +53,10 @@ def enable!
   end
 end
 
+require_relative 'web/inflector'
+
 loader = Zeitwerk::Loader.new
+
 # Make sure pro is not loaded unless Pro
 loader.ignore(Karafka::Web.gem_root.join('lib/karafka/web/ui/pro'))
 
@@ -62,9 +65,10 @@ def enable!
   loader = Zeitwerk::Loader.new
 end
 
-root = File.expand_path('..', __dir__)
 loader.tag = 'karafka-web'
-loader.inflector = Zeitwerk::GemInflector.new("#{root}/karafka/web.rb")
+# Use our custom inflector to support migrations
+root = File.expand_path('..', __dir__)
+loader.inflector = Karafka::Web::Inflector.new("#{root}/karafka/web.rb")
 loader.push_dir(root)
 
 loader.setup

lib/karafka/web/config.rb

@@ -62,6 +62,7 @@ class Config
           setting :listeners, default: [
             Tracking::Consumers::Listeners::Status.new,
             Tracking::Consumers::Listeners::Errors.new,
+            Tracking::Consumers::Listeners::Connections.new,
             Tracking::Consumers::Listeners::Statistics.new,
             Tracking::Consumers::Listeners::Pausing.new,
             Tracking::Consumers::Listeners::Processing.new,
@@ -117,9 +118,16 @@ class Config
           Karafka.env.production? ? 60_000 * 5 : 5_000
         )
 
-        # Should we display internal topics of Kafka. The once starting with `__`
-        # By default we do not display them as they are not usable from regular users perspective
-        setting :show_internal_topics, default: false
+        setting :visibility do
+          # Allows to manage visibility of payload, headers and message key in the UI
+          # In some cases you may want to limit what is being displayed due to the type of data you
+          # are dealing with
+          setting :filter, default: Ui::Models::VisibilityFilter.new
+
+          # Should we display internal topics of Kafka. The once starting with `__`
+          # By default we do not display them as they are not usable from regular users perspective
+          setting :internal_topics, default: false
+        end
 
         # How many elements should we display on pages that support pagination
         setting :per_page, default: 25

lib/karafka/web/contracts/config.rb

@@ -52,10 +52,13 @@ class Config < Web::Contracts::Base
             required(:secret) { |val| val.is_a?(String) && val.length >= 64 }
           end
 
-          required(:show_internal_topics) { |val| [true, false].include?(val) }
           required(:cache) { |val| !val.nil? }
           required(:per_page) { |val| val.is_a?(Integer) && val >= 1 && val <= 100 }
-          required(:visibility_filter) { |val| !val.nil? }
+
+          nested(:visibility) do
+            required(:filter) { |val| !val.nil? }
+            required(:internal_topics) { |val| [true, false].include?(val) }
+          end
         end
       end
     end

lib/karafka/web/errors.rb

@@ -11,6 +11,18 @@ module Errors
       # This should never happen and if you see this, please open an issue.
       ContractError = Class.new(BaseError)
 
+      # Errors specific to management
+      module Management
+        # Similar to processing error with the same name, it is raised when a critical
+        # incompatibility is detected.
+        #
+        # This error is raised when there was an attempt to operate on aggregated Web UI states
+        # that are already in a newer version that the one in the current process. We prevent
+        # this from happening not to corrupt the data. Please upgrade all the Web UI consumers to
+        # the same version
+        IncompatibleSchemaError = Class.new(BaseError)
+      end
+
       # Processing related errors namespace
       module Processing
         # Raised when we try to process reports but we do not have the current state bootstrapped

lib/karafka/web/inflector.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+module Karafka
+  module Web
+    # Web UI Zeitwerk Inflector that allows us to have time prefixed files with migrations, similar
+    # to how Rails does that.
+    class Inflector < Zeitwerk::GemInflector
+      # Checks if given path is a migration one
+      MIGRATION_ABSPATH_REGEXP = /migrations\/[0-9]+_(.*)/
+
+      # Checks if it is a migration file
+      MIGRATION_BASENAME_REGEXP = /\A[0-9]+_(.*)/
+
+      private_constant :MIGRATION_ABSPATH_REGEXP, :MIGRATION_BASENAME_REGEXP
+
+      # @param [String] basename of the file to be loaded
+      # @param abspath [String] absolute path of the file to be loaded
+      # @return [String] Constant name to be used for given file
+      def camelize(basename, abspath)
+        # If not migration directory with proper migration files, use defaults
+        return super unless abspath.match?(MIGRATION_ABSPATH_REGEXP)
+        # If base name is not of a proper name in migrations, use defaults
+        return super unless basename.match?(MIGRATION_BASENAME_REGEXP)
+
+        super(
+          # Extract only the name without the timestamp
+          basename.match(MIGRATION_BASENAME_REGEXP).to_a.last,
+          abspath
+        )
+      end
+    end
+  end
+end

lib/karafka/web/installer.rb

@@ -17,11 +17,14 @@ def install(replication_factor: 1)
         puts
         puts 'Creating necessary topics and populating state data...'
         puts
-        Management::CreateTopics.new.call(replication_factor)
+        Management::Actions::CreateTopics.new.call(replication_factor)
         wait_for_topics
-        Management::CreateInitialStates.new.call
+        Management::Actions::CreateInitialStates.new.call
         puts
-        Management::ExtendBootFile.new.call
+        puts 'Running data migrations...'
+        Management::Actions::MigrateStatesData.new.call
+        puts
+        Management::Actions::ExtendBootFile.new.call
         puts
         puts("Installation #{green('completed')}. Have fun!")
         puts
@@ -35,9 +38,12 @@ def migrate(replication_factor: 1)
         puts
         puts 'Creating necessary topics and populating state data...'
         puts
-        Management::CreateTopics.new.call(replication_factor)
+        Management::Actions::CreateTopics.new.call(replication_factor)
         wait_for_topics
-        Management::CreateInitialStates.new.call
+        Management::Actions::CreateInitialStates.new.call
+        puts
+        puts 'Running data migrations...'
+        Management::Actions::MigrateStatesData.new.call
         puts
         puts("Migration #{green('completed')}. Have fun!")
         puts
@@ -49,11 +55,14 @@ def reset(replication_factor: 1)
         puts
         puts 'Resetting Karafka Web UI...'
         puts
-        Management::DeleteTopics.new.call
+        Management::Actions::DeleteTopics.new.call
         puts
-        Management::CreateTopics.new.call(replication_factor)
+        Management::Actions::CreateTopics.new.call(replication_factor)
         wait_for_topics
-        Management::CreateInitialStates.new.call
+        Management::Actions::CreateInitialStates.new.call
+        puts
+        puts 'Running data migrations...'
+        Management::Actions::MigrateStatesData.new.call
         puts
         puts("Resetting #{green('completed')}. Have fun!")
         puts
@@ -64,16 +73,16 @@ def uninstall
         puts
         puts 'Uninstalling Karafka Web UI...'
         puts
-        Management::DeleteTopics.new.call
-        Management::CleanBootFile.new.call
+        Management::Actions::DeleteTopics.new.call
+        Management::Actions::CleanBootFile.new.call
         puts
         puts("Uninstalling #{green('completed')}. Goodbye!")
         puts
       end
 
       # Enables the Web-UI in the karafka app. Sets up needed routes and listeners.
       def enable!
-        Management::Enable.new.call
+        Management::Actions::Enable.new.call
       end
 
       private

lib/karafka/web/management/actions/base.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+module Karafka
+  module Web
+    module Management
+      # Namespace for all the commands related to management of the Web-UI in the context of
+      # Karafka. It includes things like installing, creating needed topics, etc.
+      module Actions
+        # Base class for all the commands that we use to manage
+        class Base
+          include ::Karafka::Helpers::Colorize
+
+          private
+
+          # @return [String] green colored word "successfully"
+          def successfully
+            green('successfully')
+          end
+
+          # @return [String] green colored word "already"
+          def already
+            green('already')
+          end
+
+          # @return [Array<String>] topics available in the cluster
+          def existing_topics_names
+            @existing_topics_names ||= ::Karafka::Admin
+                                       .cluster_info
+                                       .topics
+                                       .map { |topic| topic[:topic_name] }
+          end
+        end
+      end
+    end
+  end
+end

lib/karafka/web/management/actions/clean_boot_file.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+module Karafka
+  module Web
+    module Management
+      module Actions
+        # Cleans the boot file from Karafka Web-UI details.
+        class CleanBootFile < Base
+          # Web-UI enabled code
+          ENABLER_CODE = ExtendBootFile::ENABLER_CODE
+
+          private_constant :ENABLER_CODE
+
+          # Removes the Web-UI boot file data
+          def call
+            karafka_rb = File.readlines(Karafka.boot_file)
+
+            if karafka_rb.any? { |line| line.include?(ENABLER_CODE) }
+              puts 'Updating the Karafka boot file...'
+              karafka_rb.delete_if { |line| line.include?(ENABLER_CODE) }
+
+              File.write(Karafka.boot_file, karafka_rb.join)
+              puts "Karafka boot file #{successfully} updated."
+              puts 'Make sure to remove configuration and other customizations as well.'
+            else
+              puts 'Karafka Web UI components not found in the boot file.'
+            end
+          end
+        end
+      end
+    end
+  end
+end