Skip to content

Commit

Permalink
rename method and overhaul readme and changelog
Browse files Browse the repository at this point in the history
  • Loading branch information
scytherswings committed Nov 21, 2018
1 parent e3c6b94 commit f65e0dc
Show file tree
Hide file tree
Showing 5 changed files with 57 additions and 54 deletions.
17 changes: 8 additions & 9 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,12 @@
All notable changes to this project will be documented in this file.
This project adheres to [Semantic Versioning](http://semver.org/).

## [v0.2.0] : 2018-09-17
### Changelog added

### Added
- Support for overriding configuration parameters. Currently only the `:logger` option is supported.
- [DatabaseConnection](lib/pgdice/database_connection.rb) now accepts an opts hash
- [PartitionHelper](lib/pgdice/partition_helper.rb) now accepts an opts hash
- [PartitionManager](lib/pgdice/partition_manager.rb) now accepts an opts hash
- [PgSliceManager](lib/pgdice/pg_slice_manager.rb) now accepts an opts hash
- [Validation](lib/pgdice/validation.rb) now accepts an opts hash
## [v0.2.1] : 2018-10-21
### Changes
- Renamed `PartitionManager.list_batched_droppable_partitions` to
`PartitionManager.list_droppable_partitions_by_batch_size`
- Readme updated

## [v0.2.0] : 2018-10-21
- Changelog added
88 changes: 46 additions & 42 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ maintainers and any affiliated parties CANNOT BE HELD LIABLE FOR DATA LOSS OR LO
See the [LICENSE](LICENSE) for more information.


## Installation
# Installation

Add this line to your application's Gemfile:

Expand All @@ -41,9 +41,9 @@ Or install it yourself as:

$ gem install pgdice

## Usage
# Usage

### Configuration
## Configuration

You must configure `PgDice` before you can use it, otherwise you won't be able to perform any manipulation actions
on tables.
Expand All @@ -67,7 +67,7 @@ end
```


#### Configuration Parameters
### Configuration Parameters

- `database_url` - Required: The postgres database url to connect to.
- This is required since `pgslice` requires a postgres `url`.
Expand All @@ -81,15 +81,15 @@ end

- `dry_run` - Optional: Boolean value to control whether changes are executed on the database.
- Defaults to `false`
- `true` will make PgDice log out the commands but not execute them.
- `true` will make PgDice log out the commands but not execute them.

- `batch_size` - Optional: Maximum number of tables you can drop in one `drop_old_partitions` call.
- Defaults to 7.
- Keep in mind the size of your tables, drop operations are done in one command. Large tables
will take longer to drop per table and could time out if there is activity on the parent table.


#### Advanced Configuration Parameters
### Advanced Configuration Parameters

All of the following parameters are optional and honestly you probably will never need to mess with these.

Expand All @@ -99,15 +99,15 @@ All of the following parameters are optional and honestly you probably will neve
so this feature may not be very useful if you are trying to only use one connection for this utility.


### Approved Tables Configuration
## Approved Tables Configuration

In order to maintain the correct number of partitions over time you must configure a
[PgDice::Table](lib/pgdice/table.rb).

An example configuration file has been provided at [config.yml](examples/config.yml) if you would rather
declare your `approved_tables` in yaml.

#### Alternative Approved Tables Configuration
### Alternative Approved Tables Configuration

If you want to declare your [PgDice::ApprovedTables](lib/pgdice/approved_tables.rb) in your configuration
block instead, you can build them like so:
Expand All @@ -130,15 +130,16 @@ end
It is possible to use both the configuration block and a file if you so choose.
The block will take precedence over the values in the file.

### Converting existing tables to partitioned tables
## Converting existing tables to partitioned tables

__This should only be used on smallish tables and ONLY after you have tested it on a non-production copy of your
production database.__
In fact, you should just not do this in production. Schedule downtime or something and run it a few times on
a copy of your database. Then practice restoring your database some more.


This command will convert an existing table into 61 partitioned tables (30 past, 30 future, and one for today).
This command will convert the existing `comments` table into 98 partitioned tables
(90 past, 7 future, and one for today).

For more information on what's going on in the background see
[https://github.com/ankane/pgslice](https://github.com/ankane/pgslice)
Expand All @@ -148,45 +149,42 @@ For more information on what's going on in the background see
PgDice.partition_helper.partition_table('comments')
```

If you mess up (again you shouldn't use this in production). These two methods are useful for writing tests
that work with partitions.

#### Notes on partition_table
### Notes on partition_table

- You can override values configured in the `PgDice::Table` by passing them in as a hash.
- For example if you wanted to create `30` future tables instead of the configured `7` for the `comments` table
you could pass in `future: 30`.


If you mess up, again you shouldn't use this in production, you can call:

```ruby
PgDice.partition_helper.undo_partitioning!('comments')
```

#### Notes on `partition_table`

- In `partition_helper` there are versions of the methods that will throw exceptions (ending in `!`) and others
that will return a truthy value or `false` if there is a failure.
This method will revert the changes made by partitioning a table. Don't rely on this
in production if you mess up; you need to test everything thoroughly.

- `period` can be set to one of these values: `:day`, `:month`, `:year`

## Maintaining partitioned tables

### Maintaining partitioned tables

#### Adding more tables
### Adding more tables

If you have existing tables that need to periodically have more tables added you can run:

```ruby
PgDice.partition_manager.add_new_partitions('comments')
```

##### Notes on `add_new_partitions`
#### Notes on `add_new_partitions`

- The above command would add `7` new tables and their associated indexes all based on the `period` that the
partitioned table was defined with.
- The example `comments` table we have been using was configured to always keep `7` future partitions above.
- The above command would add up to `7` new tables and their associated indexes all based on the `period`
that the partitioned table was defined with.
- The example `comments` table we have been using was configured to always keep `7` future partitions above.


#### Listing droppable partitions
### Listing droppable partitions

Sometimes you just want to know what's out there and if there are tables ready to be dropped.

Expand All @@ -195,12 +193,20 @@ To list all eligible tables for dropping you can run:
PgDice.partition_manager.list_droppable_partitions('comments')
```

##### Notes on `list_droppable_partitions`
If you want to know _exactly_ which partitions will be dropped you can call:
```ruby
PgDice.partition_manager.list_droppable_partitions_by_batch_size('comments')
```

This method will show partitions that are within the configured `batch_size`.

- This method uses the `past` value from the `PgDice::Table` to determine which tables are eligible for dropping.

#### Notes on `list_droppable_partitions`

- This method uses the `past` value from the `PgDice::Table` to determine which tables are eligible for dropping.
- Like most commands, if you want to override the values it will use to check just pass them in.

#### Dropping old tables
### Dropping old tables

_Dropping tables is irreversible! Do this at your own risk!!_

Expand All @@ -210,32 +216,31 @@ If you want to drop old tables (after backing them up of course) you can run:
PgDice.partition_manager.drop_old_partitions(table_name: 'comments')
```

##### Notes on `drop_old_partitions`
#### Notes on `drop_old_partitions`

- The above example command would drop partitions that exceed the configured `past` table count
for the `PgDice::Table`.
- The example `comments` table has been configured with `past: 90` tables.
So if there were 100 tables older than `today` it would drop up to `batch_size` tables.


#### Validating everything is still working
# Validation

If you've got background jobs creating and dropping tables you're going to want to
ensure they are actually working correctly.

To validate that your expected number of tables exist, you can run:
```ruby
PgDice.validation.assert_tables('comments', future: 7, past: 90)
PgDice.validation.assert_tables('comments')
```

An [InsufficientTablesError](lib/pgdice.rb) will be raised if any conditions are not met.

This will check that the table 30 days from now exists and that there is
still a table from 90 days ago. The above example assumes the table was partitioned
by day.
This will check that there are 7 future tables from now and that there are 90 past tables
per our configuration above.


## FAQ
# FAQ

1. How do I get a postgres url if I'm running in Rails?
```ruby
Expand All @@ -257,19 +262,18 @@ end
## Planned Features

1. Full `PG::Connection` support (no more database URLs).
1. Custom schema support for all operations. Defaults to `public` currently.
1. Non time-range based partitioning.
1. Non time-range based partitioning. [PgParty](https://github.com/rkrage/pg_party) might be a good option!


## Development
# Development

After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests.
You can also run `bin/console` for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run `bundle exec rake install`.


### Running tests
## Running tests

You're going to need to have postgres 10 or greater installed.

Expand All @@ -288,12 +292,12 @@ to be a safe, welcoming space for collaboration, and contributors are expected t
the [Contributor Covenant](http://contributor-covenant.org) code of conduct.


## License
# License

The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).


## Code of Conduct
# Code of Conduct

Everyone interacting in the Pgdice project’s codebases, issue trackers, chat rooms and mailing lists is expected
to follow the [code of conduct](https://github.com/IlluminusLimited/pgdice/blob/master/CODE_OF_CONDUCT.md).
2 changes: 1 addition & 1 deletion lib/pgdice/partition_manager.rb
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ def list_droppable_partitions(table_name, params = {})
droppable_partitions(all_params)
end

def list_batched_droppable_partitions(table_name, params = {})
def list_droppable_partitions_by_batch_size(table_name, params = {})
all_params = approved_tables.smash(table_name, params)
validation.validate_parameters(all_params)
droppable_tables = batched_droppable_partitions(all_params)
Expand Down
2 changes: 1 addition & 1 deletion lib/pgdice/version.rb
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# frozen_string_literal: true

module PgDice
VERSION = '0.2.0'
VERSION = '0.2.1'
end
2 changes: 1 addition & 1 deletion test/pgdice/partition_manager_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ def test_list_partitions
end

def test_list_batched_droppable_partitions
assert_equal generate_tables.first(4), @partition_manager.list_batched_droppable_partitions('comments')
assert_equal generate_tables.first(4), @partition_manager.list_droppable_partitions_by_batch_size('comments')
end

private
Expand Down

0 comments on commit f65e0dc

Please sign in to comment.