Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Import axon as an additional internal project, support JSONB fields in Changeset and expand demo #9

Merged
merged 33 commits into from
May 29, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
efdc763
chore: Import axon from commit 553527c
perangel Apr 14, 2020
458d5a7
chore: Add axon dependencies
13rac1 Apr 14, 2020
f06083a
feat: Support JSON/B fields in Changesets
13rac1 Apr 16, 2020
ee11232
feat: Expand Demo to sync data to target DB with axon
13rac1 Apr 16, 2020
9f23ed3
feat: Additional detail to SQL error messages
13rac1 Apr 16, 2020
6198f5e
feat: Block Target DB versions less than 9.5
13rac1 Apr 16, 2020
9e989ac
feat: Support empty array/slice fields in Changeset
13rac1 Apr 16, 2020
f570d62
fix: prepareDeleteQuery will find query args in changeset.OldValues
13rac1 Apr 16, 2020
b375b66
feat: Update SERIAL sequence after INSERT queries
13rac1 Apr 18, 2020
885aeb5
feat: Add sourceDB connection and differentiate targetDB
13rac1 Apr 20, 2020
8f94419
feat: Update orphaned sequences on every INSERT
13rac1 Apr 21, 2020
0a73e83
feat: Changeset count on start, move Signal handler, print args on err
13rac1 Apr 22, 2020
c3ad1af
fix: Handle changesets including arrays of strings
13rac1 Apr 22, 2020
dc2d02a
fixup! feat: Changeset count on start, move Signal handler, print arg…
13rac1 May 27, 2020
fce1fde
fixup! feat: Update SERIAL sequence after INSERT queries
13rac1 May 27, 2020
0f5da22
fixup! feat: Block Target DB versions less than 9.5
13rac1 May 27, 2020
3b5cf68
fixup! feat: Block Target DB versions less than 9.5
13rac1 May 27, 2020
1aa6b53
fixup! feat: Changeset count on start, move Signal handler, print arg…
13rac1 May 27, 2020
c7d7a3c
fixup! feat: Update SERIAL sequence after INSERT queries
13rac1 May 27, 2020
201c16f
fixup! feat: Update SERIAL sequence after INSERT queries
13rac1 May 27, 2020
aa3e175
fixup! feat: Update SERIAL sequence after INSERT queries
13rac1 May 27, 2020
7d186ba
fixup! feat: Update SERIAL sequence after INSERT queries
13rac1 May 27, 2020
8c6f48e
fixup! feat: Update orphaned sequences on every INSERT
13rac1 May 27, 2020
734ea16
fixup! feat: Update orphaned sequences on every INSERT
13rac1 May 27, 2020
a7e5fd2
fixup! feat: Update orphaned sequences on every INSERT
13rac1 May 27, 2020
438f6ad
fixup! feat: Update orphaned sequences on every INSERT
13rac1 May 27, 2020
3f7044e
fixup! feat: Update orphaned sequences on every INSERT
13rac1 May 27, 2020
343361c
fixup! feat: Update orphaned sequences on every INSERT
13rac1 May 27, 2020
145d83a
fixup! feat: Update orphaned sequences on every INSERT
13rac1 May 27, 2020
e51e722
fixup! feat: Update orphaned sequences on every INSERT
13rac1 May 27, 2020
808fc72
fixup! feat: Additional detail to SQL error messages
13rac1 May 27, 2020
0f22315
fixup! feat: Additional detail to SQL error messages
13rac1 May 27, 2020
7dc5d18
fixup! feat: Update orphaned sequences on every INSERT
13rac1 May 27, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions cmd/axon/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,11 @@ func main() {
logger.WithError(err).Fatal("unable to connect to target database")
}

err = checkTargetVersion(targetDBConn)
if err != nil {
logger.WithError(err).Fatal("unable to use to target database")
13rac1 marked this conversation as resolved.
Show resolved Hide resolved
}

err = loadPrimaryKeys(targetDBConn)
if err != nil {
logger.WithError(err).Fatal("unable to load target DB primary keys")
Expand Down
36 changes: 32 additions & 4 deletions cmd/axon/schema.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@ package main

import (
"fmt"
"log"
"strconv"
"strings"

"github.com/jmoiron/sqlx"
"github.com/lib/pq"
Expand All @@ -11,17 +14,42 @@ import (
// maps primary key columns by table
var primaryKeys = make(map[string][]string)

func checkTargetVersion(conn *sqlx.DB) error {
var serverVersion string
err := conn.QueryRow("SHOW server_version;").Scan(&serverVersion)
if err != nil {
return err
}

version := strings.Split(serverVersion, ".")
// TODO: More thorough version checks
major, err := strconv.Atoi(version[0])
if err != nil {
return err
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these could be panics, or have some context added to the messages.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely. There's a lot of UX to add. I'll add this as a todo.

}
minor, err := strconv.Atoi(version[1])
if err != nil {
return err
}
if major == 9 && minor < 5 {
perangel marked this conversation as resolved.
Show resolved Hide resolved
// <9.5 does not support ON CONFLICT used in row update.
return fmt.Errorf("Target DB Unsupported Version: %s", serverVersion)
}
log.Printf("Target DB Version: %s", serverVersion)
return nil
}

func loadPrimaryKeys(conn *sqlx.DB) error {
var rows []struct {
TableName string `db:"table_name"`
PrimaryKey pq.StringArray `db:"primary_key"`
}
err := conn.Select(&rows, `
SELECT
t.table_name,
SELECT
t.table_name,
string_to_array(string_agg(c.column_name, ','), ',') AS primary_key
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's been a while since I wrote this, but iirc this query is slightly different for other versions of PG (older, I believe). Just making note because i can't remember if its the meta-data tables or the method for going making the array that I had to modify at some point.

FROM information_schema.key_column_usage AS c
LEFT JOIN information_schema.table_constraints AS t
FROM information_schema.key_column_usage AS c
LEFT JOIN information_schema.table_constraints AS t
ON c.constraint_name = t.constraint_name
AND t.constraint_type = 'PRIMARY KEY'
WHERE t.table_name != ''
Expand Down