diff --git a/README.md b/README.md index 37f5f361..6a6017b5 100644 --- a/README.md +++ b/README.md @@ -91,7 +91,8 @@ hierarchies of dependent views. Scenic offers a `replace_view` schema statement, resulting in a `CREATE OR REPLACE VIEW` SQL query which will update the supplied view in place, retaining -all dependencies. Materialized views cannot be replaced in this fashion. +all dependencies. Materialized views cannot be replaced in this fashion, though +the `side_by_side` update strategy may yield similar results (see below). You can generate a migration that uses the `replace_view` schema statement by passing the `--replace` option to the `scenic:view` generator: @@ -137,7 +138,7 @@ end ``` Scenic even provides a `scenic:model` generator that is a superset of -`scenic:view`. It will act identically to the Rails `model` generator except +`scenic:view`. It will act identically to the Rails `model` generator except that it will create a Scenic view migration rather than a table migration. There is no special base class or mixin needed. If desired, any code the model @@ -185,6 +186,44 @@ you would need to refresh view B first, then right after refresh view A. If you would like this cascading refresh of materialized views, set `cascade: true` when you refresh your materialized view. +## Can I update the definition of a materialized view without dropping it? + +No, but Scenic can help you approximate this behavior with its `side_by_side` +update strategy. + +Generally, changing the definition of a materialized view requires dropping it +and recreating it, either without data or with a non-concurrent refresh. The +materialized view will be locked for selects during the refresh process, which +can cause problems in your application if the refresh is not fast. + +The `side_by_side` update strategy prepares the new version of the view under a +temporary name. This includes copying the indexes from the original view and +refreshing the data. Once prepared, the original view is dropped and the new +view is renamed to the original view's name. This process minimizes the time the +view is locked for selects at the cost of additional disk space. + +You can generate a migration that uses the `side_by_side` strategy by passing +the `--side-by-side` option to the `scenic:view` generator: + +```sh +$ rails generate scenic:view search_results --materialized --side-by-side + create db/views/search_results_v02.sql + create db/migrate/[TIMESTAMP]_update_search_results_to_version_2.rb +``` + +The migration will look something like this: + +```ruby +class UpdateSearchResultsToVersion2 < ActiveRecord::Migration + def change + update_view :search_results, + version: 2, + revert_to_version: 1, + materialized: { side_by_side: true } + end +end +``` + ## I don't need this view anymore. Make it go away. Scenic gives you `drop_view` too: @@ -234,7 +273,7 @@ It's our experience that maintaining a library effectively requires regular use of its features. We're not in a good position to support MySQL, SQLite or other database users. -Scenic *does* support configuring different database adapters and should be +Scenic _does_ support configuring different database adapters and should be extendable with adapter libraries. If you implement such an adapter, we're happy to review and link to it. We're also happy to make changes that would better accommodate adapter gems. @@ -242,10 +281,10 @@ accommodate adapter gems. We are aware of the following existing adapter libraries for Scenic which may meet your needs: -* [`scenic_sqlite_adapter`]() -* [`scenic-mysql_adapter`]() -* [`scenic-sqlserver-adapter`]() -* [`scenic-oracle_adapter`]() +- [`scenic_sqlite_adapter`](https://github.com/pdebelak/scenic_sqlite_adapter) +- [`scenic-mysql_adapter`](https://github.com/EmpaticoOrg/scenic-mysql_adapter) +- [`scenic-sqlserver-adapter`](https://github.com/ClickMechanic/scenic_sqlserver_adapter) +- [`scenic-oracle_adapter`](https://github.com/cdinger/scenic-oracle_adapter) Please note that the maintainers of Scenic make no assertions about the quality or security of the above adapters. @@ -255,26 +294,24 @@ quality or security of the above adapters. ### Used By Scenic is used by some popular open source Rails apps: -[Mastodon](), -[Code.org](), and -[Lobste.rs](). +[Mastodon](https://github.com/mastodon/mastodon/), +[Code.org](https://github.com/code-dot-org/code-dot-org), and +[Lobste.rs](https://github.com/lobsters/lobsters/). ### Related projects -- [`fx`]() Versioned database functions and +- [`fx`](https://github.com/teoljungberg/fx) Versioned database functions and triggers for Rails - ### Media Here are a few posts we've seen discussing Scenic: -- [Announcing Scenic - Versioned Database Views for Rails]() by Derek Prior for thoughtbot -- [Effectively Using Materialized Views in Ruby on Rails]() by Leigh Halliday for pganalyze -- [Optimizing String Concatenation in Ruby on Rails]() -- [Materialized Views In Ruby On Rails With Scenic]() by Dawid Karczewski for Ideamotive -- [Using Scenic and SQL views to aggregate data]() by André Perdigão for Redlight Software - +- [Announcing Scenic - Versioned Database Views for Rails](https://thoughtbot.com/blog/announcing-scenic--versioned-database-views-for-rails) by Derek Prior for thoughtbot +- [Effectively Using Materialized Views in Ruby on Rails](https://pganalyze.com/blog/materialized-views-ruby-rails) by Leigh Halliday for pganalyze +- [Optimizing String Concatenation in Ruby on Rails](https://dev.to/pimp_my_ruby/from-slow-to-lightning-fast-optimizing-string-concatenation-in-ruby-on-rails-28nk) +- [Materialized Views In Ruby On Rails With Scenic](https://www.ideamotive.co/blog/materialized-views-ruby-rails-scenic) by Dawid Karczewski for Ideamotive +- [Using Scenic and SQL views to aggregate data](https://dev.to/weareredlight/using-scenic-and-sql-views-to-aggregate-data-226k) by André Perdigão for Redlight Software ### Maintainers diff --git a/lib/generators/scenic/materializable.rb b/lib/generators/scenic/materializable.rb index 7e7a6cdf..b6078237 100644 --- a/lib/generators/scenic/materializable.rb +++ b/lib/generators/scenic/materializable.rb @@ -14,7 +14,14 @@ module Materializable type: :boolean, required: false, desc: "Adds WITH NO DATA when materialized view creates/updates", - default: false + default: false, + aliases: ["--no-data"] + class_option :side_by_side, + type: :boolean, + required: false, + desc: "Uses side-by-side strategy to update materialized view", + default: false, + aliases: ["--side-by-side"] class_option :replace, type: :boolean, required: false, @@ -35,6 +42,25 @@ def replace_view? def no_data? options[:no_data] end + + def side_by_side? + options[:side_by_side] + end + + def materialized_view_update_options + set_options = {no_data: no_data?, side_by_side: side_by_side?} + .select { |_, v| v } + + if set_options.empty? + "true" + else + string_options = set_options.reduce("") do |memo, (key, value)| + memo + "#{key}: #{value}, " + end + + "{ #{string_options.chomp(", ")} }" + end + end end end end diff --git a/lib/generators/scenic/view/templates/db/migrate/update_view.erb b/lib/generators/scenic/view/templates/db/migrate/update_view.erb index 906baefd..87c94bdb 100644 --- a/lib/generators/scenic/view/templates/db/migrate/update_view.erb +++ b/lib/generators/scenic/view/templates/db/migrate/update_view.erb @@ -5,7 +5,7 @@ class <%= migration_class_name %> < <%= activerecord_migration_class %> <%= method_name %> <%= formatted_plural_name %>, version: <%= version %>, revert_to_version: <%= previous_version %>, - materialized: <%= no_data? ? "{ no_data: true }" : true %> + materialized: <%= materialized_view_update_options %> <%- else -%> <%= method_name %> <%= formatted_plural_name %>, version: <%= version %>, revert_to_version: <%= previous_version %> <%- end -%> diff --git a/lib/scenic/adapters/postgres.rb b/lib/scenic/adapters/postgres.rb index b59efdca..b9cd13e3 100644 --- a/lib/scenic/adapters/postgres.rb +++ b/lib/scenic/adapters/postgres.rb @@ -4,6 +4,10 @@ require_relative "postgres/indexes" require_relative "postgres/views" require_relative "postgres/refresh_dependencies" +require_relative "postgres/side_by_side" +require_relative "postgres/index_creation" +require_relative "postgres/index_migration" +require_relative "postgres/temporary_name" module Scenic # Scenic database adapters. @@ -22,8 +26,6 @@ module Adapters # The methods are documented here for insight into specifics of how Scenic # integrates with Postgres and the responsibilities of {Adapters}. class Postgres - MAX_IDENTIFIER_LENGTH = 63 - # Creates an instance of the Scenic Postgres adapter. # # This is the default adapter for Scenic. Configuring it via @@ -169,17 +171,9 @@ def update_materialized_view(name, sql_definition, no_data: false, side_by_side: raise_unless_materialized_views_supported if side_by_side - session_id = Time.now.to_i - new_name = generate_name name, "new_#{session_id}" - drop_name = generate_name name, "drop_#{session_id}" - IndexReapplication.new(connection: connection).on_side_by_side( - name, new_name, session_id - ) do - create_materialized_view(new_name, sql_definition, no_data: no_data) - end - rename_materialized_view(name, drop_name) - rename_materialized_view(new_name, name) - drop_materialized_view(drop_name) + SideBySide + .new(adapter: self, name: name, definition: sql_definition) + .update else IndexReapplication.new(connection: connection).on(name) do drop_materialized_view(name) @@ -202,20 +196,6 @@ def drop_materialized_view(name) execute "DROP MATERIALIZED VIEW #{quote_table_name(name)};" end - # Renames a materialized view from {name} to {new_name} - # - # @param name The existing name of the materialized view in the database. - # @param new_name The new name to which it should be renamed - # @raise [MaterializedViewsNotSupportedError] if the version of Postgres - # in use does not support materialized views. - # - # @return [void] - def rename_materialized_view(name, new_name) - raise_unless_materialized_views_supported - execute "ALTER MATERIALIZED VIEW #{quote_table_name(name)} " \ - "RENAME TO #{quote_table_name(new_name)};" - end - # Refreshes a materialized view from its SQL schema. # # This is typically called from application code via {Scenic.database}. @@ -286,15 +266,19 @@ def populated?(name) end end + # A decorated ActiveRecord connection object with some Scenic-specific + # methods. Not intended for direct use outside of the Postgres adapter. + # + # @api private + def connection + Connection.new(connectable.connection) + end + private attr_reader :connectable delegate :execute, :quote_table_name, to: :connection - def connection - Connection.new(connectable.connection) - end - def raise_unless_materialized_views_supported unless connection.supports_materialized_views? raise MaterializedViewsNotSupportedError @@ -315,16 +299,6 @@ def refresh_dependencies_for(name, concurrently: false) concurrently: concurrently ) end - - def generate_name(base, suffix) - candidate = "#{base}_#{suffix}" - if candidate.size <= MAX_IDENTIFIER_LENGTH - candidate - else - digest_length = MAX_IDENTIFIER_LENGTH - suffix.size - 1 - "#{Digest::SHA256.hexdigest(base)[0...digest_length]}_#{suffix}" - end - end end end end diff --git a/lib/scenic/adapters/postgres/index_creation.rb b/lib/scenic/adapters/postgres/index_creation.rb new file mode 100644 index 00000000..93be1ea8 --- /dev/null +++ b/lib/scenic/adapters/postgres/index_creation.rb @@ -0,0 +1,68 @@ +module Scenic + module Adapters + class Postgres + # Used to resiliently create indexes on a materialized view. If the index + # cannot be applied to the view (e.g. the columns don't exist any longer), + # we log that information and continue rather than raising an error. It is + # left to the user to judge whether the index is necessary and recreate + # it. + # + # Used when updating a materialized view to ensure the new version has all + # apprioriate indexes. + # + # @api private + class IndexCreation + # Creates the index creation object. + # + # @param connection [Connection] The connection to execute SQL against. + # @param speaker [#say] (ActiveRecord::Migration) The object used for + # logging the results of creating indexes. + def initialize(connection:, speaker: ActiveRecord::Migration.new) + @connection = connection + @speaker = speaker + end + + # Creates the provided indexes. If an index cannot be created, it is + # logged and the process continues. + # + # @param indexes [Array] The indexes to create. + # + # @return [void] + def try_create(indexes) + Array(indexes).each(&method(:try_index_create)) + end + + private + + attr_reader :connection, :speaker + + def try_index_create(index) + success = with_savepoint(index.index_name) do + connection.execute(index.definition) + end + + if success + say "index '#{index.index_name}' on '#{index.object_name}' has been created" + else + say "index '#{index.index_name}' on '#{index.object_name}' is no longer valid and has been dropped." + end + end + + def with_savepoint(name) + connection.execute("SAVEPOINT #{name}") + yield + connection.execute("RELEASE SAVEPOINT #{name}") + true + rescue + connection.execute("ROLLBACK TO SAVEPOINT #{name}") + false + end + + def say(message) + subitem = true + speaker.say(message, subitem) + end + end + end + end +end diff --git a/lib/scenic/adapters/postgres/index_migration.rb b/lib/scenic/adapters/postgres/index_migration.rb new file mode 100644 index 00000000..877a40cf --- /dev/null +++ b/lib/scenic/adapters/postgres/index_migration.rb @@ -0,0 +1,70 @@ +module Scenic + module Adapters + class Postgres + # Used during side-by-side materialized view updates to migrate indexes + # from the original view to the new view. + # + # @api private + class IndexMigration + # Creates the index migration object. + # + # @param connection [Connection] The connection to execute SQL against. + # @param speaker [#say] (ActiveRecord::Migration) The object used for + # logging the results of migrating indexes. + def initialize(connection:, speaker: ActiveRecord::Migration.new) + @connection = connection + @speaker = speaker + end + + # Retreives the indexes on the original view, renames them to avoid + # collisions, retargets the indexes to the destination view, and then + # creates the retargeted indexes. + # + # @param from [String] The name of the original view. + # @param to [String] The name of the destination view. + # + # @return [void] + def migrate(from:, to:) + source_indexes = Indexes.new(connection: connection).on(from) + retargeted_indexes = source_indexes.map { |i| retarget(i, to: to) } + source_indexes.each(&method(:rename)) + + if source_indexes.any? + say "indexes on '#{from}' have been renamed to avoid collisions" + end + + IndexCreation + .new(connection: connection, speaker: speaker) + .try_create(retargeted_indexes) + end + + private + + attr_reader :connection, :speaker + + def retarget(index, to:) + new_definition = index.definition.sub( + /ON (.*)\.#{index.object_name}/, + 'ON \1.' + to + " " + ) + + Scenic::Index.new( + object_name: to, + index_name: index.index_name, + definition: new_definition + ) + end + + def rename(index) + temporary_name = TemporaryName.new(index.index_name).to_s + connection.rename_index(index.object_name, index.index_name, temporary_name) + end + + def say(message) + subitem = true + speaker.say(message, subitem) + end + end + end + end +end diff --git a/lib/scenic/adapters/postgres/index_reapplication.rb b/lib/scenic/adapters/postgres/index_reapplication.rb index 2c556959..7cb02fda 100644 --- a/lib/scenic/adapters/postgres/index_reapplication.rb +++ b/lib/scenic/adapters/postgres/index_reapplication.rb @@ -32,51 +32,14 @@ def on(name) yield - indexes.each(&method(:try_index_create)) - end - - def on_side_by_side(name, new_table_name, temporary_id) - indexes = Indexes.new(connection: connection).on(name) - indexes.each_with_index do |index, i| - old_name = "predrop_index_#{temporary_id}_#{i}" - connection.rename_index(name, index.index_name, old_name) - end - yield - indexes.each do |index| - try_index_create(index.with_other_object_name(new_table_name)) - end + IndexCreation + .new(connection: connection, speaker: speaker) + .try_create(indexes) end private attr_reader :connection, :speaker - - def try_index_create(index) - success = with_savepoint(index.index_name) do - connection.execute(index.definition) - end - - if success - say "index '#{index.index_name}' on '#{index.object_name}' has been recreated" - else - say "index '#{index.index_name}' on '#{index.object_name}' is no longer valid and has been dropped." - end - end - - def with_savepoint(name) - connection.execute("SAVEPOINT #{name}") - yield - connection.execute("RELEASE SAVEPOINT #{name}") - true - rescue - connection.execute("ROLLBACK TO SAVEPOINT #{name}") - false - end - - def say(message) - subitem = true - speaker.say(message, subitem) - end end end end diff --git a/lib/scenic/adapters/postgres/indexes.rb b/lib/scenic/adapters/postgres/indexes.rb index 85787374..5b4421c4 100644 --- a/lib/scenic/adapters/postgres/indexes.rb +++ b/lib/scenic/adapters/postgres/indexes.rb @@ -27,7 +27,6 @@ def indexes_on(name) SELECT t.relname as object_name, i.relname as index_name, - n.nspname as schema_name, pg_get_indexdef(d.indexrelid) AS definition FROM pg_class t INNER JOIN pg_index d ON t.oid = d.indrelid @@ -45,8 +44,7 @@ def index_from_database(result) Scenic::Index.new( object_name: result["object_name"], index_name: result["index_name"], - definition: result["definition"], - schema_name: result["schema_name"] + definition: result["definition"] ) end end diff --git a/lib/scenic/adapters/postgres/side_by_side.rb b/lib/scenic/adapters/postgres/side_by_side.rb new file mode 100644 index 00000000..47919da2 --- /dev/null +++ b/lib/scenic/adapters/postgres/side_by_side.rb @@ -0,0 +1,50 @@ +module Scenic + module Adapters + class Postgres + # Updates a view using the `side-by-side` strategy where the new view is + # created and populated under a temporary name before the existing view is + # dropped and the temporary view is renamed to the original name. + class SideBySide + def initialize(adapter:, name:, definition:, speaker: ActiveRecord::Migration.new) + @adapter = adapter + @name = name + @definition = definition + @temporary_name = TemporaryName.new(name).to_s + @speaker = speaker + end + + def update + adapter.create_materialized_view(temporary_name, definition) + say "temporary materialized view '#{temporary_name}' has been created" + + IndexMigration + .new(connection: adapter.connection, speaker: speaker) + .migrate(from: name, to: temporary_name) + + adapter.drop_materialized_view(name) + say "materialized view '#{name}' has been dropped" + + rename_materialized_view(temporary_name, name) + say "temporary materialized view '#{temporary_name}' has been renamed to '#{name}'" + end + + private + + attr_reader :adapter, :name, :definition, :temporary_name, :speaker + + def connection + adapter.connection + end + + def rename_materialized_view(from, to) + connection.execute("ALTER MATERIALIZED VIEW #{from} RENAME TO #{to}") + end + + def say(message) + subitem = true + speaker.say(message, subitem) + end + end + end + end +end diff --git a/lib/scenic/adapters/postgres/temporary_name.rb b/lib/scenic/adapters/postgres/temporary_name.rb new file mode 100644 index 00000000..51cf4012 --- /dev/null +++ b/lib/scenic/adapters/postgres/temporary_name.rb @@ -0,0 +1,34 @@ +module Scenic + module Adapters + class Postgres + # Generates a temporary object name used internally by Scenic. This is + # used during side-by-side materialized view updates to avoid naming + # collisions. The generated name is based on a SHA1 hash of the original + # which ensures we do not exceed the 63 character limit for object names. + # + # @api private + class TemporaryName + # The prefix used for all temporary names. + PREFIX = "_scenic_sbs_".freeze + + # Creates a new temporary name object. + # + # @param name [String] The original name to base the temporary name on. + def initialize(name) + @name = name + @salt = SecureRandom.hex(4) + @temporary_name = "#{PREFIX}#{Digest::SHA1.hexdigest(name + salt)}" + end + + # @return [String] The temporary name. + def to_s + temporary_name + end + + private + + attr_reader :name, :temporary_name, :salt + end + end + end +end diff --git a/lib/scenic/index.rb b/lib/scenic/index.rb index 36c3a033..0e48745f 100644 --- a/lib/scenic/index.rb +++ b/lib/scenic/index.rb @@ -22,45 +22,15 @@ class Index # "CREATE INDEX index_users_on_email ON users USING btree (email)" attr_reader :definition - # The schema under which the index is defined - # @return [String] - attr_reader :schema_name - # Returns a new instance of Index # # @param object_name [String] The name of the object that has the index # @param index_name [String] The name of the index # @param definition [String] The SQL statements that defined the index - def initialize(object_name:, index_name:, definition:, schema_name:) + def initialize(object_name:, index_name:, definition:) @object_name = object_name @index_name = index_name @definition = definition - @schema_name = schema_name - end - - # Return a new instance of Index with the definition changed to create - # the index against a different object name. - # - # @param object_name [String] The name of the object that has the index - def with_other_object_name(object_name) - type = if @definition.start_with? "CREATE UNIQUE" - "CREATE UNIQUE INDEX" - else - "CREATE INDEX" - end - old_prefix = "#{type} #{@index_name} ON #{@schema_name}.#{@object_name}" - new_prefix = "#{type} #{@index_name} ON #{@schema_name}.#{object_name}" - unless @definition.start_with? old_prefix - raise "Unhandled index definition: '#{@definition}'" - end - suffix = @definition.slice((old_prefix.size)..(@definition.size)) - tweaked_definition = new_prefix + suffix - self.class.new( - object_name: object_name, - index_name: @index_name, - schema_name: @schema_name, - definition: tweaked_definition - ) end end end diff --git a/lib/scenic/statements.rb b/lib/scenic/statements.rb index 8abede92..14e1f4a5 100644 --- a/lib/scenic/statements.rb +++ b/lib/scenic/statements.rb @@ -39,10 +39,12 @@ def create_view(name, version: nil, sql_definition: nil, materialized: false) sql_definition ||= definition(name, version) if materialized + options = materialized_options(materialized) + Scenic.database.create_materialized_view( name, sql_definition, - no_data: hash_value_or_boolean(materialized, :no_data) + no_data: options[:no_data] ) else Scenic.database.create_view(name, sql_definition) @@ -82,23 +84,23 @@ def drop_view(name, revert_to_version: nil, materialized: false) # as they are mutually exclusive. # @param revert_to_version [Fixnum] The version number to rollback to on # `rake db rollback` - # @param materialized [Boolean, Hash] True if updating a materialized view. - # Set to { no_data: true } to update materialized view without loading - # data. Set to { side_by_side: true} to update materialized view with - # fewer locks but more disk usage. Defaults to false. - # @param materialized [Boolean, Hash] Set a truthy value if updating a + # @param materialized [Boolean, Hash] True or a Hash if updating a # materialized view. - # @option materialized [Boolean] :no_data (false) Set to true to create - # materialized view without running the associated query. You will need - # to perform a non-concurrent refresh to populate with data. - # @option materialized [Boolean] :side_by_side (false) Set to true to create - # the new version under a different name and atomically swap them, - # limiting downtime at the cost of doubling disk usage. + # @option materialized [Boolean] :no_data (false) Set to true to update + # a materialized view without loading data. You will need to perform a + # refresh to populate with data. Cannot be combined with the :side_by_side + # option. + # @option materialized [Boolean] :side_by_side (false) Set to true to update + # update a materialized view using our side-by-side strategy, which will + # limit the time the view is locked at the cost of increasing disk usage. + # The view is initially updated with a temporary name and atomically + # swapped once it is successfully created with data. Cannot be combined + # with the :no_data option. # @return The database response from executing the create statement. # # @example # update_view :engagement_reports, version: 3, revert_to_version: 2 - # + # update_view :comments, version: 2, revert_to_version: 1, materialized: { side_by_side: true } def update_view(name, version: nil, sql_definition: nil, revert_to_version: nil, materialized: false) if version.blank? && sql_definition.blank? raise( @@ -117,11 +119,24 @@ def update_view(name, version: nil, sql_definition: nil, revert_to_version: nil, sql_definition ||= definition(name, version) if materialized + options = materialized_options(materialized) + + if options[:no_data] && options[:side_by_side] + raise( + ArgumentError, + "no_data and side_by_side options cannot be combined" + ) + end + + if options[:side_by_side] && !transaction_open? + raise "a transaction is required to perform a side-by-side update" + end + Scenic.database.update_materialized_view( name, sql_definition, - no_data: hash_value_or_boolean(materialized, :no_data), - side_by_side: hash_value_or_boolean(materialized, :side_by_side) + no_data: options[:no_data], + side_by_side: options[:side_by_side] ) else Scenic.database.update_view(name, sql_definition) @@ -164,11 +179,17 @@ def definition(name, version) Scenic::Definition.new(name, version).to_sql end - def hash_value_or_boolean(value, key) - if value.is_a? Hash - value.fetch(key, false) + def materialized_options(materialized) + if materialized.is_a? Hash + { + no_data: materialized.fetch(:no_data, false), + side_by_side: materialized.fetch(:side_by_side, false) + } else - false + { + no_data: false, + side_by_side: false + } end end end diff --git a/spec/acceptance/user_manages_views_spec.rb b/spec/acceptance/user_manages_views_spec.rb index 426e9220..fa106faf 100644 --- a/spec/acceptance/user_manages_views_spec.rb +++ b/spec/acceptance/user_manages_views_spec.rb @@ -45,6 +45,17 @@ verify_result "Child.take.name", "Elliot" verify_schema_contains 'add_index "children"' + successfully "rails generate scenic:view child --materialized --side-by-side" + verify_identical_view_definitions "children_v02", "children_v03" + + write_definition "children_v03", "SELECT 'Juniper'::text AS name" + successfully "rake db:migrate" + + successfully "rake db:reset" + verify_result "Child.take.name", "Juniper" + verify_schema_contains 'add_index "children"' + + successfully "rake db:rollback" successfully "rake db:rollback" successfully "rake db:rollback" successfully "rails destroy scenic:model child" diff --git a/spec/generators/scenic/view/view_generator_spec.rb b/spec/generators/scenic/view/view_generator_spec.rb index 6df79ac1..14cdc5b5 100644 --- a/spec/generators/scenic/view/view_generator_spec.rb +++ b/spec/generators/scenic/view/view_generator_spec.rb @@ -37,6 +37,32 @@ end end + it "sets the no_data option when updating a materialized view" do + with_view_definition("aired_episodes", 1, "hello") do + allow(Dir).to receive(:entries).and_return(["aired_episodes_v01.sql"]) + + run_generator ["aired_episode", "--materialized", "--no-data"] + migration = migration_file( + "db/migrate/update_aired_episodes_to_version_2.rb" + ) + expect(migration).to contain "materialized: { no_data: true }" + expect(migration).not_to contain "side_by_side" + end + end + + it "sets the side-by-side option when updating a materialized view" do + with_view_definition("aired_episodes", 1, "hello") do + allow(Dir).to receive(:entries).and_return(["aired_episodes_v01.sql"]) + + run_generator ["aired_episode", "--materialized", "--side-by-side"] + migration = migration_file( + "db/migrate/update_aired_episodes_to_version_2.rb" + ) + expect(migration).to contain "materialized: { side_by_side: true }" + expect(migration).not_to contain "no_data" + end + end + it "uses 'replace_view' instead of 'update_view' if replace flag is set" do with_view_definition("aired_episodes", 1, "hello") do allow(Dir).to receive(:entries).and_return(["aired_episodes_v01.sql"]) diff --git a/spec/scenic/adapters/postgres/index_creation_spec.rb b/spec/scenic/adapters/postgres/index_creation_spec.rb new file mode 100644 index 00000000..70d046a2 --- /dev/null +++ b/spec/scenic/adapters/postgres/index_creation_spec.rb @@ -0,0 +1,54 @@ +require "spec_helper" + +module Scenic + module Adapters + describe Postgres::IndexCreation, :db do + it "successfully recreates applicable indexes" do + create_materialized_view("hi", "SELECT 'hi' AS greeting") + speaker = DummySpeaker.new + + index = Scenic::Index.new( + object_name: "hi", + index_name: "hi_greeting_idx", + definition: "CREATE INDEX hi_greeting_idx ON hi (greeting)" + ) + + Postgres::IndexCreation + .new(connection: ActiveRecord::Base.connection, speaker: speaker) + .try_create([index]) + + expect(indexes_for("hi")).not_to be_empty + expect(speaker.messages).to include(/index 'hi_greeting_idx' .* has been created/) + end + + it "skips indexes that are not applicable" do + create_materialized_view("hi", "SELECT 'hi' AS greeting") + speaker = DummySpeaker.new + index = Scenic::Index.new( + object_name: "hi", + index_name: "hi_person_idx", + definition: "CREATE INDEX hi_person_idx ON hi (person)" + ) + + Postgres::IndexCreation + .new(connection: ActiveRecord::Base.connection, speaker: speaker) + .try_create([index]) + + expect(indexes_for("hi")).to be_empty + expect(speaker.messages).to include(/index 'hi_person_idx' .* has been dropped/) + end + end + + class DummySpeaker + attr_reader :messages + + def initialize + @messages = [] + end + + def say(message, bool = false) + @messages << message + end + end + end +end diff --git a/spec/scenic/adapters/postgres/index_migration_spec.rb b/spec/scenic/adapters/postgres/index_migration_spec.rb new file mode 100644 index 00000000..72af305f --- /dev/null +++ b/spec/scenic/adapters/postgres/index_migration_spec.rb @@ -0,0 +1,24 @@ +require "spec_helper" + +module Scenic + module Adapters + describe Postgres::IndexMigration, :db, :silence do + it "moves indexes from the old view to the new view" do + create_materialized_view("hi", "SELECT 'hi' AS greeting") + create_materialized_view("hi_temp", "SELECT 'hi' AS greeting") + add_index(:hi, :greeting, name: "hi_greeting_idx") + + Postgres::IndexMigration + .new(connection: ActiveRecord::Base.connection) + .migrate(from: "hi", to: "hi_temp") + indexes_for_original = indexes_for("hi") + indexes_for_temporary = indexes_for("hi_temp") + + expect(indexes_for_original.length).to eq 1 + expect(indexes_for_original.first.index_name).not_to eq "hi_greeting_idx" + expect(indexes_for_temporary.length).to eq 1 + expect(indexes_for_temporary.first.index_name).to eq "hi_greeting_idx" + end + end + end +end diff --git a/spec/scenic/adapters/postgres/side_by_side_spec.rb b/spec/scenic/adapters/postgres/side_by_side_spec.rb new file mode 100644 index 00000000..88fd2cbe --- /dev/null +++ b/spec/scenic/adapters/postgres/side_by_side_spec.rb @@ -0,0 +1,24 @@ +require "spec_helper" + +module Scenic + module Adapters + describe Postgres::SideBySide, :db, :silence do + it "updates the materialized view to the new version" do + adapter = Postgres.new + create_materialized_view("hi", "SELECT 'hi' AS greeting") + add_index(:hi, :greeting, name: "hi_greeting_idx") + new_definition = "SELECT 'hola' AS greeting" + + Postgres::SideBySide + .new(adapter: adapter, name: "hi", definition: new_definition) + .update + result = ar_connection.execute("SELECT * FROM hi").first["greeting"] + indexes = indexes_for("hi") + + expect(result).to eq "hola" + expect(indexes.length).to eq 1 + expect(indexes.first.index_name).to eq "hi_greeting_idx" + end + end + end +end diff --git a/spec/scenic/adapters/postgres/temporary_name_spec.rb b/spec/scenic/adapters/postgres/temporary_name_spec.rb new file mode 100644 index 00000000..dfefaa20 --- /dev/null +++ b/spec/scenic/adapters/postgres/temporary_name_spec.rb @@ -0,0 +1,23 @@ +require "spec_helper" + +module Scenic + module Adapters + describe Postgres::TemporaryName do + it "generates a temporary name based on a SHA1 hash of the original" do + name = "my_materialized_view" + + temporary_name = Postgres::TemporaryName.new(name).to_s + + expect(temporary_name).to match(/_scenic_sbs_[0-9a-f]{40}/) + end + + it "does not overflow the 63 character limit for object names" do + name = "long_view_name_" * 10 + + temporary_name = Postgres::TemporaryName.new(name).to_s + + expect(temporary_name.length).to eq 52 + end + end + end +end diff --git a/spec/scenic/adapters/postgres_spec.rb b/spec/scenic/adapters/postgres_spec.rb index 2c51c54b..f95529e1 100644 --- a/spec/scenic/adapters/postgres_spec.rb +++ b/spec/scenic/adapters/postgres_spec.rb @@ -103,30 +103,6 @@ module Adapters end end - describe "#rename_materialized_view" do - it "successfully renames a materialized view" do - adapter = Postgres.new - - adapter.create_materialized_view( - "greetings", - "SELECT text 'hi' AS greeting" - ) - adapter.rename_materialized_view("greetings", "hellos") - - expect(adapter.views.map(&:name)).to include("hellos") - end - - it "raises an exception if the version of PostgreSQL is too old" do - connection = double("Connection", supports_materialized_views?: false) - connectable = double("Connectable", connection: connection) - adapter = Postgres.new(connectable) - err = Scenic::Adapters::Postgres::MaterializedViewsNotSupportedError - - expect { adapter.rename_materialized_view("greetings", "hellos") } - .to raise_error err - end - end - describe "#refresh_materialized_view" do it "raises an exception if the version of PostgreSQL is too old" do connection = double("Connection", supports_materialized_views?: false) @@ -282,6 +258,39 @@ module Adapters expect { adapter.populated?("greetings") }.to raise_error err end end + + describe "#update_materialized_view" do + it "updates the definition of a materialized view in place" do + adapter = Postgres.new + create_materialized_view("hi", "SELECT 'hi' AS greeting") + new_definition = "SELECT 'hello' AS greeting" + + adapter.update_materialized_view("hi", new_definition) + result = adapter.connection.execute("SELECT * FROM hi").first["greeting"] + + expect(result).to eq "hello" + end + + it "updates the definition of a materialized view side by side", :silence do + adapter = Postgres.new + create_materialized_view("hi", "SELECT 'hi' AS greeting") + new_definition = "SELECT 'hello' AS greeting" + + adapter.update_materialized_view("hi", new_definition, side_by_side: true) + result = adapter.connection.execute("SELECT * FROM hi").first["greeting"] + + expect(result).to eq "hello" + end + + it "raises an exception if the version of PostgreSQL is too old" do + connection = double("Connection", supports_materialized_views?: false) + connectable = double("Connectable", connection: connection) + adapter = Postgres.new(connectable) + + expect { adapter.create_materialized_view("greetings", "select 1") } + .to raise_error Postgres::MaterializedViewsNotSupportedError + end + end end end end diff --git a/spec/scenic/command_recorder_spec.rb b/spec/scenic/command_recorder_spec.rb index a33d2bc8..419fbc26 100644 --- a/spec/scenic/command_recorder_spec.rb +++ b/spec/scenic/command_recorder_spec.rb @@ -77,6 +77,24 @@ expect { recorder.revert { recorder.update_view(*args) } } .to raise_error(ActiveRecord::IrreversibleMigration) end + + it "reverts materialized views with no_data option appropriately" do + args = [:users, {version: 2, revert_to_version: 1, materialized: {no_data: true}}] + revert_args = [:users, {version: 1, materialized: {no_data: true}}] + + recorder.revert { recorder.update_view(*args) } + + expect(recorder.commands).to eq [[:update_view, revert_args]] + end + + it "reverts materialized views with side_by_side option appropriately" do + args = [:users, {version: 2, revert_to_version: 1, materialized: {side_by_side: true}}] + revert_args = [:users, {version: 1, materialized: {side_by_side: true}}] + + recorder.revert { recorder.update_view(*args) } + + expect(recorder.commands).to eq [[:update_view, revert_args]] + end end describe "#replace_view" do diff --git a/spec/scenic/statements_spec.rb b/spec/scenic/statements_spec.rb index b62ed238..c632f4dd 100644 --- a/spec/scenic/statements_spec.rb +++ b/spec/scenic/statements_spec.rb @@ -176,6 +176,36 @@ module Scenic ) end.to raise_error ArgumentError, /cannot both be set/ end + + it "raises an error is no_data and side_by_side are both set" do + definition = instance_double("Definition", to_sql: "definition") + allow(Definition).to receive(:new) + .with(:name, 3) + .and_return(definition) + + expect do + connection.update_view( + :name, + version: 3, + materialized: {no_data: true, side_by_side: true} + ) + end.to raise_error ArgumentError, /cannot be combined/ + end + + it "raises an error if not in a transaction" do + definition = instance_double("Definition", to_sql: "definition") + allow(Definition).to receive(:new) + .with(:name, 3) + .and_return(definition) + + expect do + connection(transactions_enabled: false).update_view( + :name, + version: 3, + materialized: {side_by_side: true} + ) + end.to raise_error RuntimeError, /transaction is required/ + end end describe "replace_view" do @@ -208,8 +238,20 @@ module Scenic end end - def connection - Class.new { extend Statements } + def connection(transactions_enabled: true) + DummyConnection.new(transactions_enabled: transactions_enabled) + end + end + + class DummyConnection + include Statements + + def initialize(transactions_enabled:) + @transactions_enabled = transactions_enabled + end + + def transaction_open? + @transactions_enabled end end end diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index c0ec85d1..e76b8087 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -31,6 +31,10 @@ DatabaseCleaner.clean end + config.before(:each, silence: true) do |example| + allow_any_instance_of(ActiveRecord::Migration).to receive(:say) + end + if defined? ActiveSupport::Testing::Stream config.include ActiveSupport::Testing::Stream end diff --git a/spec/support/database_schema_helpers.rb b/spec/support/database_schema_helpers.rb index 6ac553bc..7207c197 100644 --- a/spec/support/database_schema_helpers.rb +++ b/spec/support/database_schema_helpers.rb @@ -7,4 +7,22 @@ def dump_schema(stream) ActiveRecord::SchemaDumper.dump(Search.connection, stream) end end + + def ar_connection + ActiveRecord::Base.connection + end + + def create_materialized_view(name, sql) + ar_connection.execute("CREATE MATERIALIZED VIEW #{name} AS #{sql}") + end + + def add_index(view, columns, name: nil) + ar_connection.add_index(view, columns, name: name) + end + + def indexes_for(view_name) + Scenic::Adapters::Postgres::Indexes + .new(connection: ar_connection) + .on(view_name) + end end