Cannot obtain entity data from data provider by type

[son-phamngoc]

Hello, I have a problem that need your support.

I have 2 Orion-LD instances: 1st works as context broker, and 2nd works as data provider.
Firstly, I created an entity urn:ngsi-ld:Building:office001 at data provider.

curl -X POST 'http://localhost:21026/ngsi-ld/v1/entities/' \
-H 'Content-Type: application/ld+json' \
--data-raw '{
    "id": "urn:ngsi-ld:Building:office001",
    "type": "Building",
    "category": {
        "type": "Property",
        "value": "office"
    },
    "address": {
        "type": "Property",
        "value": {
            "streetAddress": "519 Orion street",
            "addressRegion": "Region1",
            "addressLocality": "Local1",
            "postalCode": "99999"
        }
    },
    "name": {
        "type": "Property",
        "value": "Office 1"
    },
    "@context": [
        "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.6.jsonld"
    ]
}'

Next, I created a registration that data provider can provide entity type "Building" for context broker.

curl -X POST 'http://localhost:1026/ngsi-ld/v1/csourceRegistrations' \
-H 'Content-Type: application/json' \
-H 'Link: <https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.6.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
-d ' {
    "id": "urn:ngsi-ld:ContextSourceRegistration:reg001",
    "type": "ContextSourceRegistration",
    "information": [
        {
            "entities": [
                {
                    "type": "Building"
                }
            ]
        }
    ],
    "endpoint": "http://orionld-prvd2:1026",
    "mode": "exclusive",
    "operations": [
        "federationOps"
    ]
}'

After that, I queried to context broker to obtain Building entities.

curl -G -X GET 'http://localhost:1026/ngsi-ld/v1/entities/' -H 'Accept: application/ld+json' \
-H 'Link: <https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.6.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
-d 'type=Building' | jq .

I expected to obtain entity urn:ngsi-ld:Building:office001 but it gave me an empty result.

If I register a specific entity provided by data provider, I can obtain that entity by id.

curl -X POST 'http://localhost:1026/ngsi-ld/v1/csourceRegistrations' \
-H 'Content-Type: application/json' \
-H 'Link: <https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.6.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
-d ' {
    "id": "urn:ngsi-ld:ContextSourceRegistration:reg002",
    "type": "ContextSourceRegistration",
    "information": [
        {
            "entities": [
                {
                    "type": "Building", "id": "urn:ngsi-ld:Building:office001"
                }
            ]
        }
    ],
    "endpoint": "http://orionld-prvd2:1026",
    "mode": "exclusive",
    "operations": [
        "federationOps"
    ]
}'

I also can get data by registering entity with propertyNames.

curl -X POST 'http://localhost:1026/ngsi-ld/v1/csourceRegistrations' \
-H 'Content-Type: application/json' \
-H 'Link: <https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.6.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
-d ' {
    "id": "urn:ngsi-ld:ContextSourceRegistration:reg003",
    "type": "ContextSourceRegistration",
    "information": [
        {
            "entities": [
                {
                    "type": "Building",
                    "id": "urn:ngsi-ld:Building:office001"
                }
            ],
            "propertyNames": [
                "name"
            ]
        }
    ],
    "endpoint": "http://orionld-prvd2:1026",
    "mode": "inclusive",
    "operations": [
        "federationOps"
    ]
}'

The problem only occurs with registration and querying by entity type.

I tried with Scorpio and it worked well with registration and querying by entity type.
I also checked this issue #1585 and saw that Orion-LD worked with that case.
So, I think this is a bug of Orion-LD. Could you help me to confirm it?
Please tell me if I did any wrong step.

Many thanks in advance.

[kzangeli]

Hello there.
My guess is that this registration failed:

{
    "id": "urn:ngsi-ld:ContextSourceRegistration:reg001",
    "type": "ContextSourceRegistration",
    "information": [
        {
            "entities": [
                {
                    "type": "Building"
                }
            ]
        }
    ],
    "endpoint": "http://orionld-prvd2:1026",
    "mode": "exclusive",
    "operations": [
        "federationOps"
    ]
}

For Exclusive registrations you must specify the entity id AND attributes (entity type is always mandatory).
If not, it's a too wide exclusive registration and that is not allowed.
Both Scorpio and Orion-LD are supposed to give an error for that.
400 Bad Request and then more info in "title" and "detail" in the payload response body.

Make sure you check the result of the registration creation attempt.
If no error is provoked, that would be a bug.

Thanks for reporting!

[son-phamngoc]

Hello @kzangeli, thanks a lot for your quick response.

My guess is that this registration failed:
Make sure you check the result of the registration creation attempt.

I checked again with curl -v, but my registration didn't failed.
201 was returned, and the registration succeed.

* upload completely sent off: 386 out of 386 bytes
< HTTP/1.1 201 Created
< Date: Thu, 21 Nov 2024 01:14:25 GMT
< Location: /ngsi-ld/v1/csourceRegistrations/urn:ngsi-ld:ContextSourceRegistration:reg001
< Content-Length: 0

I also got registration data by querying:

curl -G -X GET 'http://localhost:1026/ngsi-ld/v1/csourceRegistrations/urn:ngsi-ld:ContextSourceRegistration:reg001' -H 'Accept: application/ld+json' -H 'Link: <https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.6.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' | jq .

{
  "@context": "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.6.jsonld",
  "id": "urn:ngsi-ld:ContextSourceRegistration:reg001",
  "type": "ContextSourceRegistration",
  "endpoint": "http://orionld-prvd2:1026",
  "information": [
    {
      "entities": [
        {
          "type": "Building"
        }
      ]
    }
  ]
}

Both Scorpio and Orion-LD are supposed to give an error for that.
400 Bad Request and then more info in "title" and "detail" in the payload response body.

I checked again with Scorpio. It also registered successfully.
I tried 2 scenarios with Scorpio:
Scenario 1: 2 Scorpio instances: 1st works as context broker, and 2nd works as data provider.
Create entity at data provider:

curl -X POST 'http://localhost:19090/ngsi-ld/v1/entities/' \
-H 'Content-Type: application/ld+json' \
--data-raw '{
    "id": "urn:ngsi-ld:Building:office001",
    "type": "Building",
    "category": {
        "type": "Property",
        "value": "office"
    },
    "address": {
        "type": "Property",
        "value": {
            "streetAddress": "519 Orion street",
            "addressRegion": "Region1",
            "addressLocality": "Local1",
            "postalCode": "99999"
        }
    },
    "name": {
        "type": "Property",
        "value": "Office 1"
    },
    "@context": [
        "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.6.jsonld"
    ]
}'

Register at context broker:

curl -X POST 'http://localhost:9090/ngsi-ld/v1/csourceRegistrations' \
-H 'Content-Type: application/json' \
-H 'Link: <https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.6.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
-d ' {
    "id": "urn:ngsi-ld:ContextSourceRegistration:reg001",
    "type": "ContextSourceRegistration",
    "information": [
        {
            "entities": [
                {
                    "type": "Building"
                }
            ]
        }
    ],
    "endpoint": "http://scorpio-cent:9090",
    "mode": "exclusive",
    "operations": [
        "federationOps"
    ]
}'

Query entity successfully:

curl -G -X GET 'http://localhost:9090/ngsi-ld/v1/entities/' -H 'Accept: application/ld+json' \
-H 'Link: <https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.6.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
-d 'type=Building' | jq .

[
  {
    "id": "urn:ngsi-ld:Building:office001",
    "type": "Building",
    "address": {
      "type": "Property",
      "value": {
        "addressLocality": "Local1",
        "addressRegion": "Region1",
        "postalCode": "99999",
        "streetAddress": "519 Orion street"
      }
    },
    "category": {
      "type": "Property",
      "value": "office"
    },
    "name": {
      "type": "Property",
      "value": "Office 1"
    },
    "@context": [
      "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.6.jsonld",
      "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.8.jsonld"
    ]
  }
]

Scenario 2: Scorpio works as context broker, and Orion-LD works as data provider.
Create entity at data provider:

curl -X POST 'http://localhost:1026/ngsi-ld/v1/entities/' \
-H 'Content-Type: application/ld+json' \
--data-raw '{
    "id": "urn:ngsi-ld:Building:office001",
    "type": "Building",
    "category": {
        "type": "Property",
        "value": "office"
    },
    "address": {
        "type": "Property",
        "value": {
            "streetAddress": "519 Orion street",
            "addressRegion": "Region1",
            "addressLocality": "Local1",
            "postalCode": "99999"
        }
    },
    "name": {
        "type": "Property",
        "value": "Office 1"
    },
    "@context": [
        "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.6.jsonld"
    ]
}'

Register at context broker:

curl -X POST 'http://localhost:9090/ngsi-ld/v1/csourceRegistrations' \
-H 'Content-Type: application/json' \
-H 'Link: <https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.6.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
-d ' {
    "id": "urn:ngsi-ld:ContextSourceRegistration:reg001",
    "type": "ContextSourceRegistration",
    "information": [
        {
            "entities": [
                {
                    "type": "Building"
                }
            ]
        }
    ],
    "endpoint": "http://orion:1026",
    "mode": "exclusive",
    "operations": [
        "federationOps"
    ]
}'

Query entity, receive empty array:

curl -G -X GET 'http://localhost:9090/ngsi-ld/v1/entities/' -H 'Accept: application/ld+json' \
-H 'Link: <https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.6.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
-d 'type=Building' | jq .

[]

It seems this problem only occurs when there is Orion-LD.

Could you check again?

[kzangeli]

ok, I might have an idea what is happening here.
You expèct "new behaviour" but you're most probably running the old (still default) implementation.

Add the CLI param "-experimental" to orion-ld and we'll see how it behaves.
It's a long story, in short, the entire DB handling layer has been reimplemented using the newest driver from mongodb.
But, it is only used if you ask for it, using the -experimental CLI option (or env var ORIONLD_EXPERIMENTAL=TRUE).
It's no longer "experimental", so, don't be scared about the name.

[son-phamngoc]

@kzangeli Thanks for your support.

I tried your suggestion and got the following error when registration.

{
  "type": "https://uri.etsi.org/ngsi-ld/errors/BadRequestData",
  "title": "Invalid exclusive registration",
  "detail": "information item without specifying entity id"
}

I think this is the error that you mentioned.
So, I can understand that there is a different behavior of Orion-LD and Scorpio for this registration:

• Scorpio allows that exclusive registration and allow to obtain entity data from data provider by type.
• Orion-LD does not allow that exclusive registration because it is too wide exclusive registration.

I have another question. I changed the registration to inclusive, and the registration succeed without error. However, when I query, I got empty array again.
If I have any entity at context broker, only entities at the context broker are returned. The entities created at data provider are not returned.
Is it a normal behavior? Should Orion-LD also return an error when registration, similar to exclusive registration?

[kzangeli]

Sorry, pretty swamped lately ...
So, if Scorpio allows you to create an Exclusive registration without specifying an Entity ID, that's a bug in Scorpio.
You should file an issue about it on Scorpio's github.
For an Exclusive registration, you also need to specify attributes (protpertyNames, relationshipNames). If you don't, that's another 400 Bad Request. Or should be. If not, that would be another bug.

Now, about the problem you see with Orion-LD:
Distributed operations are costly (DB lookups) and far from everybody needs the feature, so, in Orion-LD, the feature is OFF by default.
To turn it on, add the CLI params -forwarding (env var export ORIONLD_FORWARDING=TRUE).
That hopefully makes it work.
If not, let me know, right here in this issue.

[son-phamngoc]

@kzangeli Thanks for your support.
I understood the behavior of Exclusive registration.

However, for Inclusive registration, I have not understood yet.
I started both Orion-LD instances with -forwarding and -experimental params.
Similar to above scenario, I created an entity:

curl -X POST 'http://localhost:21026/ngsi-ld/v1/entities/' \
-H 'Content-Type: application/ld+json' \
--data-raw '{
    "id": "urn:ngsi-ld:Building:office001",
    "type": "Building",
    "category": {
        "type": "Property",
        "value": "office"
    },
    "address": {
        "type": "Property",
        "value": {
            "streetAddress": "519 Orion street",
            "addressRegion": "Region1",
            "addressLocality": "Local1",
            "postalCode": "99999"
        }
    },
    "name": {
        "type": "Property",
        "value": "Office 1"
    },
    "@context": [
        "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.6.jsonld"
    ]
}'

I created an Inclusive registration. The registration succeeded without error.

curl -X POST 'http://localhost:1026/ngsi-ld/v1/csourceRegistrations' \
-H 'Content-Type: application/json' \
-H 'Link: <https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.6.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
-d ' {
    "id": "urn:ngsi-ld:ContextSourceRegistration:reg001",
    "type": "ContextSourceRegistration",
    "information": [
        {
            "entities": [
                {
                    "type": "Building"
                }
            ]
        }
    ],
    "endpoint": "http://orionld-prvd2:1026",
    "mode": "exclusive",
    "operations": [
        "federationOps"
    ]
}'

Next, a query with type gave me an empty array.

curl -G -X GET 'http://localhost:1026/ngsi-ld/v1/entities/' -H 'Accept: application/ld+json' \
-H 'Link: <https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.6.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
-d 'type=Building' | jq .
[]

Could you help me to explain this case?

[son-phamngoc]

@kzangeli I have another question.
I understand that it is required to specify entity types, id and properties when registration.
Assuming that I have 10 entities (same type) at data provider, I have to make 10 registrations for them.
Now, I want to query to Context Broker to get all 10 entities.
If the data exists in Context Broker itself, it is easy to get them by querying by entity types.
However, in distributed architecture, data is at data provider and querying by types to Context Broker always return empty result.
Is there any way to get all entities of a specific type from data provider in a single query?

[kzangeli]

ok, a few things here ...
First, only entity type is mandatory in registrations. The entity id and the names of the attributes are only mandatory if your registration is "Exclusive". There's good reasons for that.
If any other type of registration, only entity type is mandatory.

The query to get those registered entities is simply a normal

GET /ngsi-ld/v1/entities?type=XXX

For this to work in Orion-LD, you need to start it with:

-forwarding -wip entityMaps

It's working but it's a complex thing, not 100% finished yet. That's why it's under the "wip" option (Work In Progress)
You probably need -experimental or -mongocOnly as well.

[kzangeli]

So, this issue got a little bit messy.
Some misunderstandings on registration modes and then doubts about distributed operations.
Is the issue still needed?

If so, I'd propose to open a new "clean" issue and close this one.
Cause, right now I'm not sure what I should do here ...