Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Beyonce]: Allow Compound Partition/Sort Keys #34

Merged
merged 5 commits into from
Jul 20, 2020

Conversation

jcarver989
Copy link
Contributor

@jcarver989 jcarver989 commented Jul 17, 2020

@cjoelrun started this work during our last hackday in: #33. This PR splits off an incremental chunk of that work so we can merge a useful feature in.

Previously, Beyonce would force you to structure your partition / sort keys as a tuple-2 composed of [prefix, modelFieldName]. But there are many use-cases in Dynamo where you'd want to have a "compound" key -- i.e. a key formed from not 1, but multiple fields on the model.

As an example, we might have:

  1. A model for SalesRep, where our pk is ["SalesRep", "$id"].
  2. An Sale model that has the same partition key as the rep it's attached to and a sort key that's like [Sale, "$country", "$state", "$city"].

The compound key in 2) would allow us to query for all sales by a rep in a given country, state or city (by using the begins_with key condition in our query operations).

Another use-case: it's common to "invert" a table's keys using a Global Secondary Index (i.e. flip the pk and sk). Using a compound key in the sk is often useful there to ensure that when you swap the keys, a ton of data doesn't wind up in a single partition. As a concrete example, we might have:

  1. An Item model with pk: ["Item", "$id"] and sk: ["Item", "$id", "$version"] (<-- note the compound key here)
  2. A Tag model with pk: ["Item", "$itemId"] and sk: ["Tag", "$name"]

So now, we can .query(...) with an item's partition key to get it + its tags. But we'd need to swap the keys in a GSI to query the other side of the relationship -- i.e. "here's a tag, give me all the items with that tag".

If we didn't use the compound key in 1) and just used : ["Item", "$version"] -- when we flipped the pk / sk in our GSI, we'd end up with massive partitions. All items with the same version number would wind up in the same partition -- which would be ... not good :).

@jcarver989 jcarver989 requested review from aboisvert, cjoelrun and a team July 17, 2020 16:44
@cjoelrun
Copy link
Contributor

Nice. Looks good.
Do I understand correctly that this allows for creating and querying a compound key, but not for querying the partial of a compound key?

@cjoelrun
Copy link
Contributor

#36 Added test in this PR to test possible issue in model generation.

Copy link

@aboisvert aboisvert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks Cameron for initiating this!

@jcarver989
Copy link
Contributor Author

@cjoelrun :

Do I understand correctly that this allows for creating and querying a compound key, but not for querying the partial of a compound key?

That's right. This PR just adds support for compound keys. But doesn't include support for formulating a partial key.

@jcarver989
Copy link
Contributor Author

#36 Added test in this PR to test possible issue in model generation.

Great, thanks will take a look.

@jcarver989
Copy link
Contributor Author

@cjoelrun -- thanks for the test. You're right in that it was failing in this PR. Fixed it now (had missed telling our blacklist to look for compound keys as well).

@jcarver989 jcarver989 merged commit 84c46ce into master Jul 20, 2020
@jcarver989 jcarver989 deleted the feature/compound-keys branch July 20, 2020 17:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants