Grouping and aggregation

[wricciot]

This PR provides experimental support for SQL queries with grouping and aggregation. These require the mixing normaliser (mixing_norm=on in the configuration file).

The result of grouping over a relation is represented as a finite map, which in Links is treated as a list of (grouping key, associated subrelation) pairs. Aggregation can then be applied groupwise to a finite map to obtain again a relation. Such Links queries are translated to SQL queries using group by and aggregates.

[wricciot]

I'm not sure why this needs to be unsafe. Under the assumption that the second argument is a finite map, isn't

fun lookupG(x,l) {
  for (ab <- l) where (fst(a) == x) b
}

equivalent?

Ah yes. It can indeed be defined as a safe operation (modulo calls to other unsafe operations (concatMap)).

[wricciot]

Ah yes. It can indeed be defined as a safe operation (modulo calls to other unsafe operations (concatMap)).

Actually, I think my reasoning was that the semantics of the two definitions is not the same for inputs that are not valid maps (more than one entry for some keys). But surely we don't care about ill-formed inputs.

[vcgalpin]

I have tested this branch with two different Links applications with mixing_norm=false which will have exercised various database interactions:

• Guide to Pharmacology DB (nontemporal)
• https://github.com/vcgalpin/links-covid-curation (temporal)

Everything typechecked and showed no errors under execution.
I have not reviewed the code. I can do more specific testing if requested.

[vcgalpin]

I have tested this branch with two different Links applications with mixing_norm=true which will have exercised various database interactions:

• Guide to Pharmacology DB (nontemporal)

Everything typechecked and showed no errors under execution (surprisingly?)

• https://github.com/vcgalpin/links-covid-curation (temporal)

Everything typechecked. There was one error :

jslib.js:1583 Uncaught Error: Fatal error: call to server returned an error. 
Details: ***: Error: File "core/query/mixingQuery.ml", 
                     line 276, characters 8-14: Assertion failed 
                     at XMLHttpRequest.<anonymous> (jslib.js:1583:17)

It seemed to relate to an unnested query of the form

query {for (x <-v- covid_data) for (y <-- subcategory) for (z <-- csv_file) 
    where ...
       orderby ...
           [(...)]  }

Removing the orderby statement had no effect on the error.
Modifying it to

query nested {for (x <-v- covid_data) for (y <-- subcategory) for (z <-- csv_file) where ... }

appears to have resolved the error. I hypothesise that the addition of "nested" means it is no longer using the new mixing code. Should such a query work under the mixing code?

I have not reviewed the code. I can do more specific testing if requested.

[vcgalpin]

Modifying it to

query nested {for (x <-v- covid_data) for (y <-- subcategory) for (z <-- csv_file) where ... }

appears to have resolved the error. I hypothesise that the addition of "nested" means it is no longer using the new mixing code. Should such a query work under the mixing code?

Further experimentation has shown that the interaction of the mixing code and temporal queries leads to the error. For example,

query { for ( x <-v- covid_data ) [vtData(x)]

gives a similar error to above and

query nested { for ( x <-v- covid_data ) [vtData(x)]

doesn't.

[vcgalpin]

In the gtopdb code, only query nested and query flat are used, which possibly explains the success of the tests with mixing_norm=true

[wricciot]

Modifying it to

query nested {for (x <-v- covid_data) for (y <-- subcategory) for (z <-- csv_file) where ... }

appears to have resolved the error. I hypothesise that the addition of "nested" means it is no longer using the new mixing code. Should such a query work under the mixing code?

Further experimentation has shown that the interaction of the mixing code and temporal queries leads to the error. For example,

query { for ( x <-v- covid_data ) [vtData(x)]

gives a similar error to above and

query nested { for ( x <-v- covid_data ) [vtData(x)]

doesn't.

Thanks! The mixing normaliser wasn't exactly handling temporal tables. I have ported Simon's code and now queries such as the ones you mentioned are handled correctly.

[wricciot]

This should now be ready to be merged.