feat(pyclient): add support for GraphQL API to 'get' method

[YpeZ]

What are the main changes you did

This PR adds functionality to the Pyclients get method to be able to retrieve data through the GraphQL API.

• create parsing function to convert a table's metadata to a GraphQL query
• implement the distinction between CSV and GraphQL API call in get
• add columns parameter to get, specifying which columns need to be returned
• add expand parameter to expand the query on reference columns to gather all columns from the referenced table
• implement query filter on GraphQL query

In addition I added/fixed the following

• added parsing method for using the get method with as_df=True (fix(Pyclient): ensure correct column types using schema metadata when processing data from CSV API #4502)
• added truncate method (feat(emx2-pyclient): Truncate table #4250)

How to test

• explain here what to do to test this (or point to unit tests)

Checklist

• updated docs in case of new feature
• added/updated tests

[YpeZ]

In my latest commit I implemented a filter on the columns of the output of get, i.e. get me the values of only these column of the table.
This leads to discrepancies between the results of the GraphQL and CSV API, as the columns of a table from the GraphQL API and the CSV API may differ. In my GraphQL query building I include REFBACKs whereas the CSV API does not.

This and my expectation that the implementation of the query_filter for GraphQL will differ significantly from the CSV one makes me consider separating these methods into a method for each API, as for the user the arguments they fill in for either format will differ significantly.

[dtroelofsprins]

Code review:

• Looks good and makes sense

Functional tests:

Truncate table

• Truncate table works fine
• Reference from other table => Referenced by Exception

Get as_df = False or not included NOT APPROVED:

• ref/ontology column values are not returned as 'country': 'US', but as 'country': {'name': 'US'}. This is very inconvenient and for sure will break existing scripts.
• Get from an empty table returns a NoneType, where I would expect an empty list with len = 0
• Tested with biobank_directory template (with demo-data), not able to fetch from Collections and CollectionFacts: Table with id 'DiseaseTypes' not found in schema 'DieuwkesTest' => DiseaseTypes is in another schema, but not referenced as an ontology, but as a ref. F.e. Biobanks has also refs to other schema, but these are all real ontologies and this table can be fetched.
• Datatypes: bools go fine
• string type columns with decimal value with zero at the end go fine
• Empty columns are not returned (which is OK)

Get as_df = True:

• Retrieve data from an empty tables returns and empty dataframe with len=0
• dataTypes => NaN in bool type columns are filled with True
• string type columns with decimal value with zero at the end still zero removed if there are no missing values in the column

Get limited amount of columns

• as_df = True => nice filter on duplicates
• as_df = False => doesn't filter on duplicates, which is fine I guess

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[YpeZ]

@dtroelofsprins

ref/ontology column values are not returned as 'country': 'US', but as 'country': {'name': 'US'}. This is very inconvenient and for sure will break existing scripts.

This has been fixed, but for ontology/ontology_array only as ontologies are only queried by their name property. It's not possible to do this for reference columns, as the primary keys to the referenced tables differ.

Get from an empty table returns a NoneType, where I would expect an empty list with len = 0

This has been fixed.