Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use subcondition of list in get_matching_values #102

Merged
merged 105 commits into from
Sep 4, 2024
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
105 commits
Select commit Hold shift + click to select a range
3f80498
remove support for under() conditions
lmcmicu Jul 24, 2024
2841c75
Use subcondition of list in get_matching_values
jamesaoverton Jul 26, 2024
4a5ede7
small reorg
lmcmicu Jul 26, 2024
abf2a10
move get_table_ddl() function to Valve API
lmcmicu Jul 27, 2024
888c6ad
only create foreign keys in the db for from() constraints on single-v…
lmcmicu Jul 27, 2024
e9417f9
validate foreign keys on individual items from columns with list() da…
lmcmicu Jul 27, 2024
751885d
shorten variable names
lmcmicu Jul 28, 2024
6025da0
always validate constraints
lmcmicu Jul 28, 2024
70bb737
Revert "always validate constraints"
lmcmicu Jul 30, 2024
896cc3e
optimizations
lmcmicu Jul 30, 2024
15e22f4
add get_value_type() to API and toolkit
lmcmicu Jul 30, 2024
9de4aa9
do not rely on database errors tables that have list columns with fro…
lmcmicu Jul 30, 2024
ece7076
do not crash on list types whose reference type has no condition, and…
lmcmicu Jul 31, 2024
8ca60b2
remove foreign key cache
lmcmicu Aug 1, 2024
0b71b08
first pass trial implementation of new caching method for validate_ro…
lmcmicu Aug 1, 2024
30a1dda
consider conflict tables when determining forbidden and allowed value…
lmcmicu Aug 2, 2024
4eb7708
accept a list of 'received values' in when validating foreign and uni…
lmcmicu Aug 2, 2024
159a5b6
check for sql type errors when collecting values for cache
lmcmicu Aug 6, 2024
6ec116b
clean up fkey_in_db()
lmcmicu Aug 6, 2024
7df1ec0
take list values into account when collecting received_values for cac…
lmcmicu Aug 6, 2024
84f4de1
fetch allowed and forbidden values less often
lmcmicu Aug 7, 2024
4be55d5
add tree children to unique cache
lmcmicu Aug 7, 2024
e495616
use vertical caching in unique validation
lmcmicu Aug 7, 2024
a001df7
tweak
lmcmicu Aug 7, 2024
cbf21e4
fix unique caching check
lmcmicu Aug 7, 2024
7303d5d
increase perf_test rows and add it to workflow
lmcmicu Aug 7, 2024
ebc7048
tweaks
lmcmicu Aug 7, 2024
7a957c1
add numeric list test
lmcmicu Aug 8, 2024
85a5486
add missing ontology files
lmcmicu Aug 8, 2024
c7c20d5
change performance and penguin test thresholds
lmcmicu Aug 8, 2024
766a5eb
small tweaks
lmcmicu Aug 9, 2024
8b1badd
remove redundant tree:child-unique error message
lmcmicu Aug 9, 2024
cb62617
remove redundant tree:child-unique error message
lmcmicu Aug 9, 2024
e32fde7
consider also tree-foreign keys in get_rows_to_update()
lmcmicu Aug 9, 2024
53614e3
update test threshold times
lmcmicu Aug 12, 2024
7cb37ac
Merge branch 'remove-under' into use-list-subcondition
lmcmicu Aug 12, 2024
3faafc6
fix typo
lmcmicu Aug 12, 2024
94b2ad0
fix handling of pipes in match() conditions
lmcmicu Aug 12, 2024
1bb7487
update tests
lmcmicu Aug 12, 2024
89c0b83
add proper link to documentation in readme
lmcmicu Aug 12, 2024
662461d
Revert "add proper link to documentation in readme"
lmcmicu Aug 12, 2024
c324722
first documentation steps
lmcmicu Aug 13, 2024
9af7fc2
test
lmcmicu Aug 13, 2024
b19bd58
test table alignment
lmcmicu Aug 13, 2024
120fe73
test table alignment
lmcmicu Aug 13, 2024
bd16268
test table alignment
lmcmicu Aug 13, 2024
8476e28
test table alignment
lmcmicu Aug 13, 2024
3a53030
test table alignment
lmcmicu Aug 13, 2024
594d2f9
add documention on table table
lmcmicu Aug 13, 2024
87f6102
test heading
lmcmicu Aug 14, 2024
631f148
test headings
lmcmicu Aug 14, 2024
f12f2b2
finish draft of table table configuration section of readme
lmcmicu Aug 14, 2024
8ae4f7b
replace backticks in readme
lmcmicu Aug 14, 2024
1e32a0c
start readme doc for table, column, datatype, and rule tables
lmcmicu Aug 14, 2024
c41eff5
mess with table formatting
lmcmicu Aug 14, 2024
acd530d
mess with table formatting
lmcmicu Aug 14, 2024
7b5c787
undo messing with table formatting
lmcmicu Aug 14, 2024
5fbcaac
add doc for column table
lmcmicu Aug 15, 2024
57306de
tweak
lmcmicu Aug 15, 2024
3f8b522
add draft of datatype table doc
lmcmicu Aug 15, 2024
b757f5c
update readme, force name of table table to be 'table'
lmcmicu Aug 18, 2024
6e4845c
formatting tweak
lmcmicu Aug 18, 2024
44ca784
formatting tweak
lmcmicu Aug 18, 2024
b962103
formatting tweak
lmcmicu Aug 18, 2024
8b1085b
tweak
lmcmicu Aug 18, 2024
892bb6a
add required datatypes to readme
lmcmicu Aug 18, 2024
7d48742
add trimmed_line and nonspace to list of required datatypes
lmcmicu Aug 18, 2024
7cbd744
add doc on guess
lmcmicu Aug 19, 2024
b2cc105
add troubleshooting section to readme
lmcmicu Aug 19, 2024
0efe9d6
add link to rule table
lmcmicu Aug 19, 2024
c077bc7
typo fixes and tweaks to readme
lmcmicu Aug 19, 2024
d2f45d5
fix typo
lmcmicu Aug 19, 2024
ca65215
add row_order to text_views, fix broken penguin test
lmcmicu Aug 29, 2024
b7710b2
add notes on path
lmcmicu Aug 29, 2024
8555e71
tweak
lmcmicu Aug 29, 2024
cea8d2c
tweak
lmcmicu Aug 29, 2024
4817043
tweak
lmcmicu Aug 29, 2024
8f30352
tweak
lmcmicu Aug 29, 2024
5e6266a
tweak
lmcmicu Aug 29, 2024
a216451
return an error if asked to save to a default non-tsv path; add commo…
lmcmicu Aug 30, 2024
87db022
allow 'save as' for non-saveable tables
lmcmicu Aug 30, 2024
eb780fb
start on validation section
lmcmicu Aug 30, 2024
dd72d9d
try mermaid flow chart
lmcmicu Aug 30, 2024
a4d0b8c
test mermaid diagram
lmcmicu Aug 30, 2024
749742e
add validation flowchart
lmcmicu Aug 30, 2024
2771fa1
tweak validation flowchart
lmcmicu Aug 30, 2024
3b12cae
tweak validation flowchart
lmcmicu Aug 30, 2024
36487ea
tweak validation flowchart
lmcmicu Aug 30, 2024
c94ada2
add headers for subsections to be done
lmcmicu Aug 30, 2024
5ea7f91
add more validation stuff to readme
lmcmicu Aug 30, 2024
878b759
add section on batch validation
lmcmicu Aug 31, 2024
a42e491
fix typo
lmcmicu Aug 31, 2024
282c4a1
add stuff to readme on editing data
lmcmicu Aug 31, 2024
fa28ee2
complete first draft of design and concepts documentation
lmcmicu Aug 31, 2024
b1b67a6
add table of contents to readme
lmcmicu Sep 2, 2024
c82b253
use better example in readme
lmcmicu Sep 2, 2024
cc1f04a
tweak
lmcmicu Sep 2, 2024
589e45c
tweak
lmcmicu Sep 2, 2024
d25716c
table formatting
lmcmicu Sep 2, 2024
25416f3
table formatting
lmcmicu Sep 2, 2024
c040aa0
table formatting
lmcmicu Sep 2, 2024
68ca403
typo fixes
lmcmicu Sep 2, 2024
4678042
typo fixes
lmcmicu Sep 2, 2024
21ea38f
small readme updates
lmcmicu Sep 3, 2024
a5c3e17
fix typos and tweaks
lmcmicu Sep 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
586 changes: 262 additions & 324 deletions src/toolkit.rs

Large diffs are not rendered by default.

269 changes: 140 additions & 129 deletions src/validate.rs

Large diffs are not rendered by default.

362 changes: 275 additions & 87 deletions src/valve.rs

Large diffs are not rendered by default.

15 changes: 10 additions & 5 deletions test/expected/messages.tsv
Original file line number Diff line number Diff line change
@@ -1,20 +1,25 @@
table row column level rule message value
table 3 type error datatype:table_type type should be a table type rule
table 12 options error option:unrecognized unrecognized option foo
table 17 options warning option:overrides overrides db_table db_view
table 17 options error option:reserved reserved for internal use internal
table 18 options warning option:overrides overrides save db_view
table 21 options warning option:overrides overrides edit no-edit
table 21 options warning option:overrides overrides save no-save
table 18 options warning option:overrides overrides db_table db_view
table 18 options error option:reserved reserved for internal use internal
table 19 options warning option:overrides overrides save db_view
table 22 options warning option:overrides overrides edit no-edit
table 22 options warning option:overrides overrides save no-save
table1 5 base error key:unique Values of base must be unique http://purl.obolibrary.org/obo/VO_
table1 5 prefix error key:primary Values of prefix must be unique VO
table1 10 base error key:unique Values of base must be unique http://www.w3.org/1999/02/22-rdf-syntax-ns#
table1 10 prefix error key:primary Values of prefix must be unique rdf
table11 3 foo error key:primary Values of foo must be unique d
table11 4 foo error key:primary Values of foo must be unique e
table11 5 foo error key:primary Values of foo must be unique e
table12 1 foo error key:foreign Value '_treacle_' of column foo is not in table13.foo _schmeichel_ _darrigol_ _treacle_
table12 2 foo error datatype:custom_b Value 'beta' of column foo should be one of a list of custom_a tokens separated by spaces _alpha_ beta _delta_ _gamma_ phi
table12 2 foo error datatype:custom_b Value 'phi' of column foo should be one of a list of custom_a tokens separated by spaces _alpha_ beta _delta_ _gamma_ phi
table12 2 foo error key:foreign Value 'beta' of column foo is not in table13.foo _alpha_ beta _delta_ _gamma_ phi
table12 2 foo error key:foreign Value 'phi' of column foo is not in table13.foo _alpha_ beta _delta_ _gamma_ phi
table12 3 bar error key:foreign Value 'plorgo' of column bar is not in table13.bar plorgo
table12 3 foo error key:foreign Value '_schmeagol_' of column foo is not in table13.foo _schmeagol_
table2 1 foo error rule:foo-2 bar cannot be null if foo is not null 5
table2 1 foo error rule:foo-4 bar must be 'y' or 'z' if foo = 5 5
table2 2 foo error rule:foo-1 bar must be null whenever foo is null
Expand Down
15 changes: 10 additions & 5 deletions test/expected/messages_a1.tsv
Original file line number Diff line number Diff line change
@@ -1,20 +1,25 @@
table cell level rule message value
table D3 error datatype:table_type type should be a table type rule
table E12 error option:unrecognized unrecognized option foo
table E17 warning option:overrides overrides db_table db_view
table E17 error option:reserved reserved for internal use internal
table E18 warning option:overrides overrides save db_view
table E21 warning option:overrides overrides edit no-edit
table E21 warning option:overrides overrides save no-save
table E18 warning option:overrides overrides db_table db_view
table E18 error option:reserved reserved for internal use internal
table E19 warning option:overrides overrides save db_view
table E22 warning option:overrides overrides edit no-edit
table E22 warning option:overrides overrides save no-save
table1 B5 error key:unique Values of base must be unique http://purl.obolibrary.org/obo/VO_
table1 A5 error key:primary Values of prefix must be unique VO
table1 B10 error key:unique Values of base must be unique http://www.w3.org/1999/02/22-rdf-syntax-ns#
table1 A10 error key:primary Values of prefix must be unique rdf
table11 D3 error key:primary Values of foo must be unique d
table11 D4 error key:primary Values of foo must be unique e
table11 D5 error key:primary Values of foo must be unique e
table12 A1 error key:foreign Value '_treacle_' of column foo is not in table13.foo _schmeichel_ _darrigol_ _treacle_
table12 A2 error datatype:custom_b Value 'beta' of column foo should be one of a list of custom_a tokens separated by spaces _alpha_ beta _delta_ _gamma_ phi
table12 A2 error datatype:custom_b Value 'phi' of column foo should be one of a list of custom_a tokens separated by spaces _alpha_ beta _delta_ _gamma_ phi
table12 A2 error key:foreign Value 'beta' of column foo is not in table13.foo _alpha_ beta _delta_ _gamma_ phi
table12 A2 error key:foreign Value 'phi' of column foo is not in table13.foo _alpha_ beta _delta_ _gamma_ phi
table12 B3 error key:foreign Value 'plorgo' of column bar is not in table13.bar plorgo
table12 A3 error key:foreign Value '_schmeagol_' of column foo is not in table13.foo _schmeagol_
table2 D1 error rule:foo-2 bar cannot be null if foo is not null 5
table2 D1 error rule:foo-4 bar must be 'y' or 'z' if foo = 5 5
table2 D2 error rule:foo-1 bar must be null whenever foo is null
Expand Down
15 changes: 10 additions & 5 deletions test/expected/messages_after_api_test.tsv
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
table row column level rule message value
table 3 type error datatype:table_type type should be a table type rule
table 12 options error option:unrecognized unrecognized option foo
table 17 options warning option:overrides overrides db_table db_view
table 17 options error option:reserved reserved for internal use internal
table 18 options warning option:overrides overrides save db_view
table 21 options warning option:overrides overrides edit no-edit
table 21 options warning option:overrides overrides save no-save
table 18 options warning option:overrides overrides db_table db_view
table 18 options error option:reserved reserved for internal use internal
table 19 options warning option:overrides overrides save db_view
table 22 options warning option:overrides overrides edit no-edit
table 22 options warning option:overrides overrides save no-save
table1 5 base error key:unique Values of base must be unique http://purl.obolibrary.org/obo/VO_
table1 5 prefix error key:primary Values of prefix must be unique VO
table1 10 base error key:unique Values of base must be unique http://www.w3.org/1999/02/22-rdf-syntax-ns#
Expand All @@ -14,8 +14,13 @@ table10 1 numeric_foreign_column error datatype:integer numeric_foreign_column s
table10 1 numeric_foreign_column error datatype:trimmed_line numeric_foreign_column should be a line of text that does not begin or end with whitespace
table11 2 foo error key:primary Values of foo must be unique d
table11 3 foo error key:primary Values of foo must be unique d
table12 1 foo error key:foreign Value '_treacle_' of column foo is not in table13.foo _schmeichel_ _darrigol_ _treacle_
table12 2 foo error datatype:custom_b Value 'beta' of column foo should be one of a list of custom_a tokens separated by spaces _alpha_ beta _delta_ _gamma_ phi
table12 2 foo error datatype:custom_b Value 'phi' of column foo should be one of a list of custom_a tokens separated by spaces _alpha_ beta _delta_ _gamma_ phi
table12 2 foo error key:foreign Value 'beta' of column foo is not in table13.foo _alpha_ beta _delta_ _gamma_ phi
table12 2 foo error key:foreign Value 'phi' of column foo is not in table13.foo _alpha_ beta _delta_ _gamma_ phi
table12 3 bar error key:foreign Value 'plorgo' of column bar is not in table13.bar plorgo
table12 3 foo error key:foreign Value '_schmeagol_' of column foo is not in table13.foo _schmeagol_
table2 1 child error key:unique Values of child must be unique b
table2 1 child error tree:child-unique Values of child must be unique b
table2 2 foo error rule:foo-1 bar must be null whenever foo is null
Expand Down
5 changes: 4 additions & 1 deletion test/src/column.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,10 @@ table11 parent text
table11 xyzzy text
table11 foo text primary
table11 bar barmatic text from(view1.bar)
table12 foo custom_b
table12 foo custom_b from(table13.foo)
table12 bar word from(table13.bar)
table13 foo text
table13 bar text
view1 foo empty trimmed_line
view1 bar empty trimmed_line
view2 foo empty trimmed_line
Expand Down
2 changes: 1 addition & 1 deletion test/src/datatype.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -23,5 +23,5 @@ text any text TEXT textarea
trimmed_line line match(/\S([^\n]*\S)*/) a line of text that does not begin or end with whitespace
trimmed_text text exclude(/^\s+|\s+$/) text that does not begin or end with whitespace
word nonspace exclude(/\W/) a single word: letters, numbers, underscore %s
custom_a text match(/_\w+_/) a series of word characters surrounded by underscores
custom_a text match(/_\w+_/) a word surrounded by underscores
custom_b text list(custom_a, ' ') a list of custom_a tokens separated by spaces
7 changes: 4 additions & 3 deletions test/src/ontology/table12.tsv
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
foo
_schmeichel_ _darrigol_ _treacle_
_alpha_ beta _delta_ _gamma_ phi
foo bar
_schmeichel_ _darrigol_ _treacle_ schmorg
_alpha_ beta _delta_ _gamma_ phi plorg
_schmeagol_ plorgo
8 changes: 8 additions & 0 deletions test/src/ontology/table13.tsv
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
foo bar
_schmeichel_ schmorg
_darrigol_ plorg
_alpha_ georg
_beta_ freorg
_delta_ dorgo
_gamma_ morgo
_phi_ slorgo
1 change: 1 addition & 0 deletions test/src/table.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ table9 test/src/ontology/table9.tsv The ninth data table
table10 test/src/ontology/table10.tsv The tenth data table
table11 test/src/ontology/table11.tsv The eleventh data table
table12 test/src/ontology/table12.tsv The twelvth data table
table13 test/src/ontology/table13.tsv The thirteenth data table
view1 test/output/view1.sql db_table db_view internal
view2 test/output/view2.sh save db_view
view3 db_view
Expand Down
Loading