Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Per environment master shard #538

Open
wants to merge 45 commits into
base: master
Choose a base branch
from

Commits on Jun 21, 2019

  1. Configuration menu
    Copy the full SHA
    74145f1 View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2019

  1. Start on setting up sample_app -- not working

    Running cucumber gives the error:
    
    cannot load such file -- test/unit (LoadError)
    calh committed Jun 25, 2019
    Configuration menu
    Copy the full SHA
    48090f0 View commit details
    Browse the repository at this point in the history
  2. Ooops, revert a couple files.

    These weren't supposed to make it into this PR
    calh committed Jun 25, 2019
    Configuration menu
    Copy the full SHA
    8e2671d View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2019

  1. Add DB2 server

    calh committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    c3ac7ca View commit details
    Browse the repository at this point in the history
  2. Trying to add ibm_db gem to gemspec

    This is failing with the stacktrace:
    
    Caused by:
    LoadError: cannot load such file -- arel/visitors/bind_visitor
    /usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require'
    /usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `block in require'
    /usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in `load_dependency'
    /usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require'
    /usr/src/app/.bundle/ruby/2.5.0/gems/ibm_db-4.0.0/lib/active_record/connection_adapters/ibm_db_adapter.rb:13:in `<top (required)>'
    /usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require'
    /usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `block in require'
    /usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in `load_dependency'
    /usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require'
    /usr/src/app/lib/octopus/proxy_config.rb:245:in `initialize_adapter'
    calh committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    ce42246 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    75a1ef5 View commit details
    Browse the repository at this point in the history
  4. Install the DB2 ODBC driver

    The gem install does this, but the download is slow.  Watching
    the wget progress is more satisfying.
    calh committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    77a3155 View commit details
    Browse the repository at this point in the history
  5. Add a DB2 shard

    calh committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    faaca64 View commit details
    Browse the repository at this point in the history
  6. Appriasal run for ibm_db

    calh committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    28e9190 View commit details
    Browse the repository at this point in the history
  7. Check that @config exists. Fixes thiagopradi#413

    Error message is:
    
    ActiveRecord::StatementInvalid: NoMethodError: undefined method `[]' for
    nil:NilClass:
    
      from octopus/lib/octopus/abstract_adapter.rb:23:in `octopus_shard'
      from octopus/lib/octopus/abstract_adapter.rb:12:in `instrument'
      from vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.11.1/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
    
    This has shown to surface in the ibm_db and sqlserver adapters.
    calh committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    3e9f50b View commit details
    Browse the repository at this point in the history
  8. Check that @config exists. Fixes thiagopradi#413

    Error message is:
    
    ActiveRecord::StatementInvalid: NoMethodError: undefined method `[]' for
    nil:NilClass:
    
      from octopus/lib/octopus/abstract_adapter.rb:23:in `octopus_shard'
      from octopus/lib/octopus/abstract_adapter.rb:12:in `instrument'
      from vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.11.1/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
    
    This has shown to surface in the ibm_db and sqlserver adapters.
    calh committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    4fd1129 View commit details
    Browse the repository at this point in the history
  9. Monkey patch IBM_DBAdapter to add @config

    This allows the InstrumenterDecorator to properly log the shard name
    calh committed Jun 26, 2019
    Configuration menu
    Copy the full SHA
    5f3bc57 View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2019

  1. Ooof, typo in filename

    calh committed Jun 28, 2019
    Configuration menu
    Copy the full SHA
    fa21f02 View commit details
    Browse the repository at this point in the history
  2. Adding db2

    calh committed Jun 28, 2019
    Configuration menu
    Copy the full SHA
    9da3a89 View commit details
    Browse the repository at this point in the history
  3. Adding DB2

    calh committed Jun 28, 2019
    Configuration menu
    Copy the full SHA
    08ad9e0 View commit details
    Browse the repository at this point in the history
  4. Setting up DB2 server image

    calh committed Jun 28, 2019
    Configuration menu
    Copy the full SHA
    ee8c9d2 View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2019

  1. Configuration menu
    Copy the full SHA
    4a488a9 View commit details
    Browse the repository at this point in the history
  2. Optionally load the ibm_db gem

    calh committed Jul 1, 2019
    Configuration menu
    Copy the full SHA
    66c61c6 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a70aad5 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    5ef3e27 View commit details
    Browse the repository at this point in the history
  5. Remove ibm_db from rails 5.1

    calh committed Jul 1, 2019
    Configuration menu
    Copy the full SHA
    82205de View commit details
    Browse the repository at this point in the history
  6. First DB2 test is working

    calh committed Jul 1, 2019
    Configuration menu
    Copy the full SHA
    1c658c1 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2019

  1. Note on DB2

    calh committed Jul 2, 2019
    Configuration menu
    Copy the full SHA
    7f48d85 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c6e267f View commit details
    Browse the repository at this point in the history
  3. Tests for DB2

    calh committed Jul 2, 2019
    Configuration menu
    Copy the full SHA
    3495359 View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2019

  1. Add db2 port/db

    calh committed Jul 3, 2019
    Configuration menu
    Copy the full SHA
    1c5b373 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    67dc844 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a96f523 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    3dcd867 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2019

  1. Skip migrations with DB2. Random bug in ibm_db.

    Running tests individually works, but running the whole
    migration_spec.rb file will recreate the issue _almost_
    every time, in a random spec.
    
    Here's the full error and stack trace:
    
    Octopus::Migration should send the query to the correct shard
         Failure/Error:
           delegate :adapter_name, :add_transaction_record, :case_sensitive_modifier,
             :type_cast, :to_sql, :quote, :quote_column_name, :quote_table_name,
             :quote_table_name_for_assignment, :supports_migrations?, :table_alias_for,
             :table_exists?, :in_clause_length, :supports_ddl_transactions?,
             :sanitize_limit, :prefetch_primary_key?, :current_database,
             :combine_bind_parameters, :empty_insert_statement_value, :assume_migrated_upto_version,
             :schema_cache, :substitute_at, :internal_string_options_for_primary_key, :lookup_cast_type_from_column,
             :supports_advisory_locks?, :get_advisory_lock, :initialize_internal_metadata_table,
             :release_advisory_lock, :prepare_binds_for_database, :cacheable_query, :column_name_for_operation,
             :prepared_statements, :transaction_state, :create_table, to: :select_connection
    
         RuntimeError:
           An unexpected error occurred during retrieval of table metadata: uncaught throw :"Fetch Failure: <error message could not be retrieved>"
         # /usr/src/ruby-ibmdb/IBM_DB_Adapter/ibm_db/lib/active_record/connection_adapters/ibm_db_adapter.rb:1858:in `rescue in tables'
         # /usr/src/ruby-ibmdb/IBM_DB_Adapter/ibm_db/lib/active_record/connection_adapters/ibm_db_adapter.rb:1845:in `tables'
         # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/schema_statements.rb:55:in `table_exists?'
         # ./lib/octopus/proxy.rb:25:in `table_exists?'
         # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/schema_migration.rb:20:in `block in table_exists?'
         # ./gemfiles/.bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/deprecation/reporting.rb:36:in `silence'
         # ./gemfiles/.bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/deprecation/instance_delegator.rb:20:in `silence'
         # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/schema_migration.rb:20:in `table_exists?'
         # /usr/src/ruby-ibmdb/IBM_DB_Adapter/ibm_db/lib/active_record/connection_adapters/ibm_db_adapter.rb:48:in `create_table'
         # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/schema_statements.rb:1008:in `initialize_schema_migrations_table'
         # ./lib/octopus/proxy.rb:200:in `initialize_schema_migrations_table'
         # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/migration.rb:1127:in `initialize'
         # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/migration.rb:1017:in `new'
         # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/migration.rb:1017:in `run'
         # ./lib/octopus/migration.rb:187:in `block in run_with_octopus'
         # ./lib/octopus/proxy.rb:86:in `block (2 levels) in run_queries_on_shard'
         # ./lib/octopus/proxy.rb:345:in `using_shard'
         # ./lib/octopus/proxy.rb:85:in `block in run_queries_on_shard'
         # ./lib/octopus/proxy.rb:329:in `keeping_connection_proxy'
         # ./lib/octopus/proxy.rb:84:in `run_queries_on_shard'
         # ./lib/octopus/proxy.rb:93:in `block in send_queries_to_multiple_shards'
         # ./lib/octopus/proxy.rb:92:in `map'
         # ./lib/octopus/proxy.rb:92:in `send_queries_to_multiple_shards'
         # ./lib/octopus/migration.rb:186:in `run_with_octopus'
         # ./spec/support/octopus_helper.rb:46:in `migrate_to_version'
         # ./spec/support/octopus_helper.rb:37:in `migrating_to_version'
         # ./spec/octopus/migration_spec.rb:86:in `block (2 levels) in <top (required)>'
         # ------------------
         # --- Caused by: ---
         # UncaughtThrowError:
         #   uncaught throw :"\xA3\xC9\x8E+H\xEA\xC9\vX\x82g\x8B\xAAU\x00\x00\bY\r\x8C\xAAU\x00\x00\x1A\xD0\xC1\xE2\xCF\xE9\x1Dd0\xB3m\x8B\xAAU\x00\x00\x80T\r\x8C\xAAU\x00\x00\x01\x99\xD1o,"
         #   /usr/src/ruby-ibmdb/IBM_DB_Adapter/ibm_db/lib/active_record/connection_adapters/ibm_db_adapter.rb:1845:in `fetch_assoc'
    calh committed Jul 18, 2019
    Configuration menu
    Copy the full SHA
    ea41105 View commit details
    Browse the repository at this point in the history
  2. DB2 doesn't have boolean types

    calh committed Jul 18, 2019
    Configuration menu
    Copy the full SHA
    091f9ef View commit details
    Browse the repository at this point in the history
  3. Adding Appriasals just for DB2.

    This keeps the test suite going for most users that won't
    be using the ibm_db gem
    calh committed Jul 18, 2019
    Configuration menu
    Copy the full SHA
    65d981d View commit details
    Browse the repository at this point in the history
  4. Ooops, wrong names

    calh committed Jul 18, 2019
    Configuration menu
    Copy the full SHA
    c63e2a0 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    86058e1 View commit details
    Browse the repository at this point in the history
  6. Merge pull request #1 from calh/docker-dev-system

    Docker dev system
    calh authored Jul 18, 2019
    Configuration menu
    Copy the full SHA
    fc18f35 View commit details
    Browse the repository at this point in the history
  7. Merge pull request #2 from calh/nil_shard_config

    Check that @config exists.  Fixes thiagopradi#413
    calh authored Jul 18, 2019
    Configuration menu
    Copy the full SHA
    cc72467 View commit details
    Browse the repository at this point in the history
  8. Merge pull request #3 from calh/ibm_db

    DB2 ibm_db gem support
    calh authored Jul 18, 2019
    Configuration menu
    Copy the full SHA
    2cbfd9e View commit details
    Browse the repository at this point in the history
  9. Issue with stock ibm_db 4.0.0 gem

    ibmdb/ruby-ibmdb#89
    
    Until that issue is officially resolved by IBM, I'm using my local
    fork for testing Octopus.
    calh committed Jul 18, 2019
    Configuration menu
    Copy the full SHA
    f241079 View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2019

  1. Fix for DB2 hanging indefinitely on disconnect with Passenger

    This is an oddball bug, deep in the DB2 C library.  Disconnecting
    from the database during the Passenger fork process causes it to
    hang indefinitely
    calh committed Jul 31, 2019
    Configuration menu
    Copy the full SHA
    9a8b293 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    8b59e83 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    98de909 View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2019

  1. Skip the connected? test with DB2

    Since we don't disconnect from DB2, this spec fails.
    calh committed Aug 5, 2019
    Configuration menu
    Copy the full SHA
    92b4caf View commit details
    Browse the repository at this point in the history
  2. Merge branch 'ibm_db'

    calh committed Aug 5, 2019
    Configuration menu
    Copy the full SHA
    d5431c2 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2019

  1. Add a per-environment master_shard option

    In many setups, the default master shard might be a different
    database server in each environment.  A single `master_shard`
    config setting doesn't fit.  This patch first checks
    the shards.yml for `octopus` => Rails env => master_shard
    for a setting, and still falls back to `octopus` => `master_shard`
    if it doesn't exist
    calh committed Aug 28, 2019
    Configuration menu
    Copy the full SHA
    a4d2a06 View commit details
    Browse the repository at this point in the history
  2. Better fix for per-environment master shards

    I cowboyed up a bit too fast on the first fix.  This moves the
    per-environment master_shard selection to its own yaml key
    calh committed Aug 28, 2019
    Configuration menu
    Copy the full SHA
    1d07220 View commit details
    Browse the repository at this point in the history