Return incorrect results when query integer numbers

[Cynthia-77]

Current Behavior

Write the number 5 of unspecified data type, int type, and long type to the database.

The Java code which writes the numbers to the database is：

Repository db = new SailRepository(new MemoryStore());
IRI john = Values.iri("http://JohnSmith");
IRI mark = Values.iri("http://Mark");
IRI peter = Values.iri("http://Peter");

try (RepositoryConnection conn = db.getConnection()) {
	conn.add(john, FOAF.AGE, Values.literal(5));
	conn.add(mark, FOAF.AGE, Values.literal(String.valueOf(5), XSD.INT));
	conn.add(peter, FOAF.AGE, Values.literal(String.valueOf(5), XSD.LONG));
}

Then query the database for the statement whose literal value is 5.

The SparQL query statement is:

SELECT *
WHERE { ?s ?p 5. }

The query result is empty, instead of the three number 5 in the database.

Expected Behavior

The expected query result is:

?s	?p
<http://JohnSmith>	<http://xmlns.com/foaf/0.1/age>
<http://Mark>	<http://xmlns.com/foaf/0.1/age>
<http://Peter>	<http://xmlns.com/foaf/0.1/age>

Steps To Reproduce

1. Write the following Java code to write the numbers to the database:

   Repository db = new SailRepository(new MemoryStore());
   IRI john = Values.iri("http://JohnSmith");
   IRI mark = Values.iri("http://Mark");
   IRI peter = Values.iri("http://Peter");
   
   try (RepositoryConnection conn = db.getConnection()) {
   	conn.add(john, FOAF.AGE, Values.literal(5));
   	conn.add(mark, FOAF.AGE, Values.literal(String.valueOf(5), XSD.INT));
   	conn.add(peter, FOAF.AGE, Values.literal(String.valueOf(5), XSD.LONG));
   }

2. Execute the following query:

   SELECT *
   WHERE { ?s ?p 5. }
   

   The corresponding Java code is:

   try (RepositoryConnection conn = db.getConnection()) {
   	String queryString = "SELECT * WHERE { ?s ?p 5. }";
   	TupleQuery query = conn.prepareTupleQuery(queryString);
   	TupleQueryResultHandler tsvWriter = new SPARQLResultsTSVWriter(System.out);
   	query.evaluate(tsvWriter);
   }

Version

4.0.0-M1

Are you interested in contributing a solution yourself?

No response

Anything else?

No response

[kenwenzel]

This is not intuitive but correct. Integer literals in SPARQL and Turtle are big integers (xsd:integer) and not xsd:int or xsd:long.
You may take a look at the spec here:
https://www.w3.org/TR/sparql11-query/#QSynLiterals

BTW, if you query for "5"^^xsd:int then you should get two results.

[hmottestad]

Does something like this work?

SELECT *
WHERE { ?s ?p ?o. FILTER(?o = 5) }

[Cynthia-77]

This is not intuitive but correct. Integer literals in SPARQL and Turtle are big integers (xsd:integer) and not xsd:int or xsd:long. You may take a look at the spec here: https://www.w3.org/TR/sparql11-query/#QSynLiterals

BTW, if you query for "5"^^xsd:int then you should get two results.

We know Integer literals in SPARQL are xsd:integer. The issue is that since users can directly write 5 (without explicitly declaring its data type) to database, why could not query it in the same way? Actually, we can get the expected result on Apache Jena.

When we query for "5"^^xsd:int, we indeed get two results. However, since xsd:int and xsd:long are subtypes of xsd:integer, we think graph database systems should support implicit data type conversion for them.

[hmottestad]

How about as part of normalization? Eg. Normalize all integers to xsd:integer and all decimals to xsd:decimal (or something like that).

https://rdf4j.org/javadoc/3.6.2/org/eclipse/rdf4j/rio/DatatypeHandler.html

[abrokenjester]

@hmottestad that somewhat stretches what is meant with normalization, I think. Datatype normalization is supposed to "clean up" the lexical representation for a particular datatype, put it in a canonical form (e.g. turn "0001" into "1" for an integer). Interpreting a long as an integer (or even a decimal) is not really normalization, it's type-casting, or inheritance/entailment.

If we were to extend support for this (note that for comparison operations we already support datatype inheritance of this form, it's just direct BGP matching that's in question here) I would prefer we support it as a form of query rewriting (either as part of a stacked sail or perhaps as an optional configurable feature in the query evaluation strategy).

[Cynthia-77]

Can you provide an example on how you insert the data with Jena? Maybe the literals are converted to xsd:integer at this point.

The following is an example of Jena:

Model model = ModelFactory.*createDefaultModel*();
Resource johnSmith = model.createResource("http://JohnSmith");
Resource peter = model.createResource("http://Peter");
Resource mark = model.createResource("http://Mark");
johnSmith.addProperty(FOAF.*phone*, model.createTypedLiteral(5));
mark.addProperty(FOAF.*phone*, model.createTypedLiteral("5", XSD.*xint*.getURI()));
peter.addProperty(FOAF.*phone*, model.createTypedLiteral("5", XSD.*xlong*.getURI()));

And this is the code querying for the integer literals:

String queryString = "SELECT * WHERE { ?s ?p 5 .}";
Query query = QueryFactory.*create*(queryString);
QueryExecution qexec = QueryExecutionFactory.*create*(query, model);
ResultSet results = qexec.execSelect();
ResultSetFormatter.*out*(System.*out*, results, query);
qexec.close();

The query result contains three solutions:

?s	?p
<http://JohnSmith>	<http://xmlns.com/foaf/0.1/phone>
<http://Mark>	<http://xmlns.com/foaf/0.1/phone>
<http://Peter>	<http://xmlns.com/foaf/0.1/phone>

We still consider that from the users' point of view, it is unfriendly for RDF4j to return such query results.

[abrokenjester]

Thanks, that confirms, I think, that what Jena does is (partial) D-entaiilment at query time.

We still consider that from the users' point of view, it is unfriendly for RDF4j to return such query results.

I wouldn't go as far as "unfriendly" (we're a pretty friendly bunch really, for the most part) but I agree it would be useful to have limited D-entailment support in RDF4J as well.

[Cynthia-77]

I wouldn't go as far as "unfriendly"

Sorry about that! I just mean it is inconvenient for users to use, not saying you unfriendly.

[Cynthia-77]

Does something like this work?

SELECT *
WHERE { ?s ?p ?o. FILTER(?o = 5) }

This SPARQL query can work. Obviously, RDF4j implicitly converts 5 to xsd:int in FILTER operation.
So is it possible to add implicit type conversion to meet users’ query need, just like it can be queried through FILTER?

[hmottestad]

How = works in filters is specified in the SPARQL 1.1 spec. Same goes for nodes in the Basic Graph Pattern (the query body). I think it would be a fair amount of work if you want to try to extend how the BGP is evaluated in RDF4J. I think it's best to just follow the specification in this case and either use a query with a FILTER or to clean your data before loading.

[abrokenjester]

Actually what RDF4J does in this case is find a common ancestor datatatype for the two arguments of the comparison. 5 is of type xsd:integer while the bindings of c are all subtypes of xsd:integer, so it ends up upcasting those bindings and comparing xsd:integers.

It would potentially be possible to do this for BGP matching as well, but it's not required by the specs, and likely to have severe performance implications.

[kenwenzel]

We could determine the smallest numerical type for a literal and store that instead of the given type. That would allow us to match BGPs. But there is a problem if the literal is queried as a wild card: What type should be returned in this case? All types according to the D-entailment?

[abrokenjester]

We could determine the smallest numerical type for a literal and store that instead of the given type. That would allow us to match BGPs. But there is a problem if the literal is queried as a wild card: What type should be returned in this case? All types according to the D-entailment?

Yeah I really don't think manipulating the data itself is the way forward here. An optional/configurable feature to support this kind of D-entailment via query rewrite (basically turning a BGP with a typed literal constant into a BGP with a variable and a filter codition, under the hood) is the most spec-compliant way to support something like this - and also something that we can implement in a way that is not store-dependent so it will immediately work on all backends. Still on the fence over whether this should be a stacked sail or a config option in the query evaluation strategy. Either way, the API option to include/exclude inferred statements could possibly be reused to tweak this behavior as well.

[abrokenjester]

I'll try and write up a feature proposal for this over the weekend. Then we can discuss if anybody has the time and resources to work on it.

[abrokenjester]

I'm on holiday so can't dig but we fixed something in #3025 related to this. Edit or actually we briefly considered it but then decided not to proceed.

[kenwenzel]

Yes, this is exactly the issue we are talking about here.
I think that the current implementation is correct in that it returns xsd:int for Java integers and xsd:integer for Java BigInteger objects.

[abrokenjester]

I've written a draft feature proposal here: #3815 .

[afs]

@Cynthia-77

we can get the expected result on Apache Jena.

Presumably, that is with TDB1.

The Jena API does the same as RDF4J - Java int 5 becomes "5"^^xsd:int.

In TDB1 (only) "5"^^xsd:int, and all derived types of xsd:integer, will be stored as "5"^^xsd:integer. The original datatype xsd:int is lost. Some users, with a knowledge management perspective, prefer retaining the original datatype and use the general approach of a FILTER. Jena isn't the only system to do this. Losing "nonNegativeInteger" etc is unfortunate.

TDB2 retains the datatype; it still canonicalises the lexical form "+5", "005" all become "5".

[Cynthia-77]

This is the query results:

?s	?p
<http://JohnSmith>	<http://xmlns.com/foaf/0.1/phone>
<http://Mark>	<http://xmlns.com/foaf/0.1/phone>
<http://Peter>	<http://xmlns.com/foaf/0.1/phone>

It can get all the number 5 in the database, no matter what datatype it is.

[hmottestad]

That's interesting. Your first query shows the datatypes are the same as in RDF4J, but when processing the query it doesn't seem that Jena is as strict as RDF4J.

https://www.w3.org/TR/sparql11-query/#matchNumber

From my understanding of the spec a number without a datatype in a sparql query is implicitly an xsd:integer unless it is a decimal.

[Cynthia-77]

So I wonder if the query results like Jena are more in line with the general thinking of users and more convenient for users.

[hmottestad]

Andy Seaborne (the Jena project lead) says that Jena doesn't implement this support across all the backends in Jena. It's only something that works in one particular backend. I get the feeling that it should be considered a bug, but because people have gotten accustomed to it then it's hard to fix/remove.

There is a community working group that's working on SPARQL 1.2, but I'm afraid there aren't any companies that have stepped up to fund their work. I think SPARQL 1.2 would definitely be the place to create a good solution. I know there is also interest in units of measure, I could definitely see some overlap between the need for querying across numeric datatypes and querying across something like temperature.

[afs]

general thinking of users
There are lots of users and lots of user tasks.

Sometimes details matter; losing the exact lexical form may not matter that much though in possible with, say, messy data.
Datatype is more problematic.

Intuitions like substitution (match a variable, use that variable in another match) break down if both value and term matching are in effect.

Or CONSTRUCTWHERE { ?s ?p 5 } is now not a subgraph of the original data.

In tasks such as SHACL validation, datatype does matter (xsd:positiveInteger).

[afs]

It depends on the storage: with an in-memory, plain, non-trandactional model, there is a value-matching in the storage itself. This is the only storage that does this.