Problem with subscriptions when orion container crashes or stops.

[AizenvoltPrime]

services:
    orion:
        labels:
            org.fiware: "tutorial"
        image: fiware/orion-ld:${ORION_LD_VERSION}
        hostname: orion
        container_name: fiware-orion
        depends_on:
            - mongo-db
        networks:
            - default
        ports:
            - "${ORION_LD_PORT}:${ORION_LD_PORT}"
        command: -dbhost mongo-db -logLevel DEBUG -forwarding -mongocOnly -wip entityMaps
        healthcheck:
            test: curl --fail -s http://orion:${ORION_LD_PORT}/version || exit 1
            interval: 5s

    # Quantum Leap is persisting Short Term History to Crate-DB
    quantumleap:
        image: orchestracities/quantumleap:${QUANTUMLEAP_VERSION}
        labels:
            org.fiware: "tutorial"
        hostname: quantumleap
        container_name: fiware-quantumleap
        ports:
            - "${QUANTUMLEAP_PORT}:${QUANTUMLEAP_PORT}"
        depends_on:
            - crate-db
            - redis-db
        environment:
            - CRATE_HOST=crate-db
            - REDIS_HOST=redis-db
            - REDIS_PORT=${REDIS_PORT}
            - REDIS_USERNAME=${REDIS_USERNAME}
            - REDIS_PASSWORD=${REDIS_PASSWORD}
            - LOGLEVEL=DEBUG
        healthcheck:
            test: curl --fail -s http://quantumleap:${QUANTUMLEAP_PORT}/version || exit 1

    # Databases
    mongo-db:
        labels:
            org.fiware: "tutorial"
        image: mongo:${MONGO_DB_VERSION}
        hostname: mongo-db
        container_name: db-mongo
        expose:
            - "${MONGO_DB_PORT}"
        ports:
            - "${MONGO_DB_PORT}:${MONGO_DB_PORT}" # localhost:27017
        networks:
            - default
        volumes:
            - mongo-db:/data
        healthcheck:
            test:
                [
                    "CMD",
                    "mongo",
                    "--quiet",
                    "127.0.0.1/test",
                    "--eval",
                    "'quit(db.runCommand({ ping: 1 }).ok ? 0 : 2)'",
                ]
            interval: 5s

    crate-db:
        labels:
            org.fiware: "tutorial"
        image: crate:${CRATE_VERSION}
        hostname: crate-db
        container_name: db-crate
        networks:
            - default
        ports:
            # Admin UI
            - "4200:4200"
            # Transport protocol
            - "4300:4300"
        command: crate -Cauth.host_based.enabled=false  -Ccluster.name=democluster -Chttp.cors.enabled=true -Chttp.cors.allow-origin="*" -Cdiscovery.type=single-node
        environment:
            - CRATE_HEAP_SIZE=2g # see https://crate.io/docs/crate/howtos/en/latest/deployment/containers/docker.html#troubleshooting
        volumes:
            - crate-db:/data

    redis-db:
        labels:
            org.fiware: "tutorial"
        image: redis:${REDIS_VERSION}
        hostname: redis-db
        container_name: db-redis
        networks:
            - default
        ports:
            - "${REDIS_PORT}:${REDIS_PORT}"
        volumes:
            - redis-db:/data
            - ./redis.conf:/usr/local/etc/redis/redis.conf
        command: redis-server /usr/local/etc/redis/redis.conf
        environment:
            - REDIS_USERNAME=${REDIS_USERNAME}
            - REDIS_PASSWORD=${REDIS_PASSWORD}
        healthcheck:
            test: >
                sh -c 'export REDISCLI_AUTH="${REDIS_PASSWORD}" &&
                redis-cli -h redis-db -p ${REDIS_PORT} --user ${REDIS_USERNAME} ping'
            interval: 10s


networks:
    default:
        labels:
            org.fiware: "tutorial"
        ipam:
            config:
                - subnet: 172.18.1.0/24

volumes:
    mongo-db: ~
    crate-db: ~
    redis-db: ~

I have the above docker compose. The problem is that if orion container crashes or stops and I start it again the subscriptions made to quantumleap even though they still exist in mongo db, they don't work and when I update entity data only the orion entity data is updated and the new data doesn't get added to crate db. I have to recreate the subscriptions manually in order for them to work again. Is there any solution to this or am I doing something wrong with the setup?

[kzangeli]

ok, that's not good.
The subscriptions are kept in a cache in RAM, apart from being persisted in mongodb.
To match entity updates with subscriptions, the cache is used, so it seems like you've found a bug in the subscription cache.
A bit weird as I implemented the cache perhaps 10 years ago and I've hardly touched it since ...

Add the information of the creation of your subscriptions to this issue and let me know exactly what you do in order to see the problem.
Then I can try to reproduce the problem. If I'm able to reproduce, it's typically quite easy to fix it.

[AizenvoltPrime]

entity creation example:

curl --location 'http://SERVER_IP:1026/ngsi-ld/v1/entities/' \
--header 'Content-Type: application/ld+json' \
--header 'NGSILD-Tenant: openiot' \
--data-raw '{
  "id": "urn:ngsi-ld:test_entity_id",
  "type": "test_entity_type",
  "temperature": {
    "type": "Property",
    "value": 23.4
  },
  "@context": [
    "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.8.jsonld",
    {
      "temperature": "https://mysite.com/api/ngsi-ld-attributes/temperature"
    }
  ]
}'

entity subscription:

curl --location 'http://SERVER_IP:1026/ngsi-ld/v1/subscriptions/' \
--header 'Content-Type: application/ld+json' \
--header 'NGSILD-Tenant: openiot' \
--data-raw '{
  "description": "Notify QuantumLeap of changes to test_entity_type",
  "type": "Subscription",
  "entities": [
    {
      "id": "urn:ngsi-ld:test_entity_id",
      "type": "test_entity_type"
    }
  ],
  "watchedAttributes": ["temperature"],
  "notification": {
    "attributes": ["temperature"],
    "format": "normalized",
    "endpoint": {
      "uri": "http://quantumleap:8668/v2/notify",
      "accept": "application/json",
      "receiverInfo": [
        {
          "key": "fiware-service",
          "value": "openiot"
        }
      ]
    }
  },
  "@context": [
    "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.8.jsonld",
    {
      "temperature": "https://mysite.com/api/ngsi-ld-attributes/temperature"
    }
  ]
}'

adding data to crate db:

curl --location 'http://SERVER_IP:1026/ngsi-ld/v1/entities/urn:ngsi-ld:test_entity_id/attrs' \
--header 'Content-Type: application/ld+json' \
--header 'NGSILD-Tenant: openiot' \
--data-raw '{
    "temperature": {
        "type": "Property",
        "value": 8.5
    },
    "@context": [
        "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.8.jsonld",
        {
            "temperature": "https://mysite.com/api/ngsi-ld-attributes/temperature"
        }
    ]
}
'

if you do that it will create the entity and its subscription and add some data to it in the crate db. After that stop the orion ld container and start it again. If you try to add new data to the entity in crate db nothing will be added.

Also for the variables in docker compose:
ORION_LD_PORT=1026
ORION_LD_VERSION=1.7.0
QUANTUMLEAP_VERSION=1.0.0
QUANTUMLEAP_PORT=8668
CRATE_VERSION=latest
MONGO_DB_PORT=27017
MONGO_DB_VERSION=latest
REDIS_PORT=6380
REDIS_VERSION=6
for REDIS_USERNAME and REDIS_PASSWORD put whatever

[kzangeli]

ok, good. That I can work with.
One comment though.
I see you are giving a core context as part of your user context.
The broker already has the core context "builtin".
By default it's v1.6.1 but this can be changed using the CLI param -coreContext (I need to update that, it doesn't include the 1.8.1 core context).

Giving a core context to the broker as a user context is not a very good idea.
I'd propose to either give just:

"@context": {
  "temperature": "https://mysite.com/api/ngsi-ld-attributes/temperature"
}

Or, even better host your user context somewhere and just give the URL to it.

[AizenvoltPrime]

Ok, thanks for the info I will take that into account and adjust the entity creation process.

[kzangeli]

So, I found the error. I explain:
As you are creating the subscription with a compound @context, the broker needs to create and host this context.
This new context is hosted and added to the cache and all is working just fine.
However, it seems I have forgotten to persist the new context in the database, so when the broker is restarted, the context is gone!!!
Important bug of mine. I feel a little bit ashamed to tell you the truth.
As the context is done, your attribute "temperature" will no longer be expanded the same way and thus, notifications will not get triggered - QL will receive nothing after the restart.

So, if you follow my advice and stop using "application/ld+json", the complication with the complex @context disappears and your problem is gone.

1. Host your context somewhere (Orion-LD offers that service, but most people use github or some web-server)
2. Use the Link header to pass the context to the broker, and always "application/json", both for Content-Type and Accept.

I will of course proceed and fix my bug.

[kzangeli]

I just fixed another issue, closely related to this one: #1715.
Might be you'll see some difference

[AizenvoltPrime]

Tested it just now. The problem remains.

[kzangeli]

ok, pity ... :)

Sorry for geing a bit AWOL lately.
Got some issue reports from one of the European projects I work in.
That's where my salary comes from so, I need to give those guys priority.
Mid next week I should be able to get back with you again.

[AizenvoltPrime]

No problem. I am actually surprised by the interest you showed regarding the issues I am facing, you don't see that commitment often. Thank you for all your efforts.

[kzangeli]

ok, in an issue coming from elsewhere, I just fixed a bug about repopulating the subscription cache on restart.
Please try again and let me know.

[AizenvoltPrime]

Should I just pull again with latest tag?

[kzangeli]

Yes, just don't leave "latest" in there. It's dangerous as "latest" keeps changing.

[AizenvoltPrime]

Just checked again. The problem remains.

[kzangeli]

ok, thanks.
Let's "divide and conquer" a little, might shed some light.
Do the same test, but without using any user context. Just the broker's builtin core context and only application/json, no link headers,
The result of a test like that would certainly help.

[AizenvoltPrime]

It works without @context and without link in the header

[kzangeli]

Good.
So, did you follow that advice I gave you about not giving me the core context as part of your user context?
That is ALWAYS a bad idea. Especially as the version of the core context of the broker is subject to change.

Also, avoid at all cost to use "inline" contexts. Also a very bad idea.
Use always application/json in Content-Type + Accept headers and the Link header.

My guess is you already tried that.
If not, please try that.

The best approach is to host your user context somewhere and then give that URL in the Link header.
The broker will download your user context and add it to its internal context cache, the very first time.
After that, the URL is known and ready, it will just be found in the brokers context cache.
You save time in execution and possible complications.

[AizenvoltPrime]

I tried that solution with link you mentioned but it has several problems with backslashes in url and duplicate attrNames and more that I mentioned in detail in the other issue I had opened and we had discussed. So I still use @context.

[kzangeli]

Ah, that, yeah I remember.
That's beyond weird. I looked into that, in the source code.
No backslashes inserted, as far I as I could see (I also don't remember implementing anything like that).
Please paste again the URL and the complete context here and I'll take another look.
Nobody else has seen any issue like that, so, quite weird!

[AizenvoltPrime]

Here are my requests:

Create entity:

curl --location 'http://SERVER_IP:1026/ngsi-ld/v1/entities/' \
--header 'Accept: application/json'\
--header 'Content-Type: application/json' \
--header 'NGSILD-Tenant: openiot' \
--header 'Link: <https://mysite.com/api/ngsi-ld-attributes>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
--data '{
  "id": "urn:ngsi-ld:test_entity_id",
  "type": "test_entity_type",
  "temperature": {
    "type": "Property",
    "value": 23.4
  }
}'

Create subscription:

curl --location 'http://SERVER_IP:1026/ngsi-ld/v1/subscriptions/' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--header 'NGSILD-Tenant: openiot' \
--header 'Link: <https://mysite.com/api/ngsi-ld-attributes>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
--data '{
  "description": "Notify QuantumLeap of changes to test_entity_type",
  "type": "Subscription",
  "entities": [
    {
      "id": "urn:ngsi-ld:test_entity_id",
      "type": "test_entity_type"
    }
  ],
  "watchedAttributes": ["temperature"],
  "notification": {
    "attributes": ["temperature"],
    "format": "normalized",
    "endpoint": {
      "uri": "http://quantumleap:8668/v2/notify",
      "accept": "application/json",
      "receiverInfo": [
        {
          "key": "fiware-service",
          "value": "openiot"
        }
      ]
    }
  }
}'

the entity in mongo is saved like:

{
  "_id": {
    "id": "urn:ngsi-ld:test_entity_id",
    "type": "https://uri.etsi.org/ngsi-ld/default-context/test_entity_type",
    "servicePath": "/"
  },
  "attrNames": [
    "https:\\/\\/mysite.com\\/api\\/ngsi-ld-attributes\\/temperature"
  ],
  "attrs": {
    "https:\\/\\/mysite=com\\/api\\/ngsi-ld-attributes\\/temperature": {
      "type": "Property",
      "creDate": 1733158754.530026,
      "modDate": 1733158754.530026,
      "value": 23.4,
      "mdNames": []
    }
  },
  "creDate": 1733158754.530026,
  "modDate": 1733158754.530026,
  "lastCorrelator": ""
}

The problems start after I try to patch:

curl --location --request PATCH 'http://SERVER_IP:1026/ngsi-ld/v1/entities/urn:ngsi-ld:test_entity_id/' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--header 'NGSILD-Tenant: openiot' \
--header 'Link: <https://mysite.com/api/ngsi-ld-attributes>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
--data '{
    "temperature": {
        "type": "Property",
        "value": 3.5
    }
}'

the mongo entity changes to:

{
  "_id": {
    "id": "urn:ngsi-ld:test_entity_id",
    "type": "https://uri.etsi.org/ngsi-ld/default-context/test_entity_type",
    "servicePath": "/"
  },
  "attrNames": [
    "https:\\/\\/mysite.com\\/api\\/ngsi-ld-attributes\\/temperature",
    "https:\\/\\/mysite.com\\/api\\/ngsi-ld-attributes\\/temperature"
  ],
  "attrs": {
    "https:\\/\\/mysite=com\\/api\\/ngsi-ld-attributes\\/temperature": {
      "type": "Property",
      "creDate": 1733159022.6124334,
      "modDate": 1733159022.6124334,
      "value": 12.5,
      "mdNames": []
    }
  },
  "creDate": 1733158754.530026,
  "modDate": 1733159022.6124334,
  "lastCorrelator": ""
}

notice how the attr names became 2:

 "https:\\/\\/mysite.com\\/api\\/ngsi-ld-attributes\\/temperature",
 "https:\\/\\/mysite.com\\/api\\/ngsi-ld-attributes\\/temperature"

Also if I patch again they will become 3 attr names instead of remaining one unique for each attribute like it happened with the old method of creating entities. I just wanted to replace with the latest value.

The context is like this:

{
    "@context": {
        "myAttributes": "https://mysite.com/api/ngsi-ld-attributes/",
        "temperature": {
            "@id": "myAttributes:temperature",
            "@type": "xsd:float",
            "unit": "°C",
            "en": "Temperature",
            "el": "Θερμοκρασία"
        },
        "humidity": {
            "@id": "myAttributes:humidity",
            "@type": "xsd:integer",
            "unit": "%",
            "en": "Humidity",
            "el": "Υγρασία"
        }
     }
  }

Also regarding the patch request I mentioned. Is creating duplicate entries in "attrNames": [ an intented functionality every time a patch request is made?
Another question about the

curl --location --request DELETE 'http://SERVER_IP:1026/ngsi-ld/v1/jsonldContexts/fdabaae4-b098-11ef-b78e-0242ac120106?reload=true'

I noticed and you mentioned it as well that the mongo entity of context doesnt change from the original one after running the delete request. The attributes that were added to the context after the initial load are expanded correctly after I update the context in the site page and run the delete request. The problem I have is since the mongo db entity for context doesnt change that means if orion container stops or is deleted the attributes that were added to the context after the initial download wont be included in the context anymore and if I try to create a new entity with those attributes without reloading the context again they wont have the correct url. Is there a way to force orion to recreate the context entity in mongo as well as refreshing its context cache? If there isn't I guess I have to go back to the previous way of creating entities without link because my context is dynamic since users in my app can add attributes to the context when they create a new entity that has an attribute that doesnt exist in the context. The old method didn't have any of these problems like the backslashes in link, the duplicate entries in attrNames array that get added every time on patch and also the problem with the context that there isn't a way to recreate the mongo entity after initial download and that means that if orion container stops then the additional context is lost and needs to be reloaded manually,

[kzangeli]

Ok, good, thanks for the refresh.
Dynamic context ... that's a first.
The problem with duplicates of the attribute has no doubt to do with the context changing, I'm close to 100% sure about that.
I see you use the broker to host the context, so, perhaps those unwanted backslashes are fruit of the reload of the context. I'll run a test on this and I'd like you to make sure those backslashes don't come from the remote context.
Pretty sure the error is there, the updated context somehow get additional slashes.

It's actually not duplicates of one attribute. They're different attributes as the long name of the attribute (expanded from short name using the context) changes.

[AizenvoltPrime]

I am glad that dynamic context is a first cause we plan on getting the certification powered by FIWARE soon and its good to know that nobody has done something like this before, as far you know at least. In regards to the context it remained the same while I did the above test. I noticed something very weird in mongo orionld database though. I dont have time now, but I will tell you later about it. Its extremely weird.

[kzangeli]

So, that functional test I was going to implement about reloading the cached context, looking for a similar functest I actually found one doing exactly that.

Take a look at this test:
https://github.com/FIWARE/context.Orion-LD/blob/develop/test/functionalTest/cases/0000_ngsild/ngsild_context-delete-and-reload.test

I am now inclined to think that the bug is elsewhere.
I'm thinking when you re-upload your modified user context to https://mysite.com/api/ngsi-ld-attributes, perhaps those backslashes enter?

Actually, I saw something really weird just now in your messages.

  "attrNames": [
    "https:\\/\\/mysite.com\\/api\\/ngsi-ld-attributes\\/temperature",
    "https:\\/\\/mysite.com\\/api\\/ngsi-ld-attributes\\/temperature"
  ],
  "attrs": {
    "https:\\/\\/iemis-demo=indigital=gr\\/api\\/ngsi-ld-attributes\\/temperature": {

If we overlook the backslashes, what's in the attrNames array is supposed to be a copy of what's "toplevel" inside 'attrs'.
The keys in mongo can¡t have dots. That's forbidden (obviously, as mongo uses the dot as separator), so in the DB we change all dots in keys to '=' (and '=' is forbidden in a longname).

The attrNames array is a "hack" an ex-colleague of mine from Telefonica invented (I don't like it), to find attributes quicker in a DB lookup. No need to replace those dots for rqual signs as the dots are in the RHS, so slightly quicker lookups.

Now, if an attribute has the modified name "http://a=b=c/q19", in attrNames, the corresponding name must be "http://a.b.c/q19".
But that's not the case!!!
How on earth did we get here???

Also, both array fields have the exact same name.
I imagine I do the lookup inside "attrs", can't find it, and add it to "attrNames".

What a freaking mess!!!
I would need to be able to reproduce this.
It's a complete disaster!!!

We talk later

[AizenvoltPrime]

  "attrNames": [
    "https:\\/\\/mysite.com\\/api\\/ngsi-ld-attributes\\/temperature",
    "https:\\/\\/mysite.com\\/api\\/ngsi-ld-attributes\\/temperature"
  ],
  "attrs": {
    "https:\\/\\/iemis-demo=indigital=gr\\/api\\/ngsi-ld-attributes\\/temperature": {

In regards to this I forgot to replace in the example I gave you the real site name with mysite.com

so I should have wrote:

  "attrNames": [
    "https:\\/\\/mysite.com\\/api\\/ngsi-ld-attributes\\/temperature",
    "https:\\/\\/mysite.com\\/api\\/ngsi-ld-attributes\\/temperature"
  ],
  "attrs": {
    "https:\\/\\/mysite=com\\/api\\/ngsi-ld-attributes\\/temperature": {

If I understood correctly, you said in attrNames no values should have "."
Even in the old way with @context they still have . in attrNames. I don't know what's causing this. I even tried to run the containers local on a fresh environment and I still get the same problems with backslashes and double attrNames. If you have any questions, please don't hesitate to ask. I will help find the problem in any way I can. In regards to the weird thing in orionld database I mentioned I don't know what caused it, many entities were created there but I deleted them then deleted the orionld database in mongo then deleted the orion container and started the process from the start and they weren't created again. Maybe the reload that I had tried in the past caused that problem, I am not sure. If it happens again I will let you know.

The main issues right now, as I understand them are the . in attrNames, the duplicate attrNames, and the backslashes.

[kzangeli]

so I should have wrote:

ok, now it makes more sense. Thank God !!! :)))

If I understood correctly, you said in attrNames no values should have "."
No, the other way around.
The keys, the left-hand-side (in this case "attrNames" itself) cannot have any dots. The RHS (right-hand-side) can have dots, that's fine.

As an example, look at the object that is the id of an entity (old stuff coming from NGSIv1, but orion-ld is backward compatible and uses the very same DB model):

  "_id": {
    "id": "urn:ngsi-ld:test_entity_id",
    "type": "https://uri.etsi.org/ngsi-ld/default-context/test_entity_type",
    "servicePath": "/"
  },
' ' ' 

To query mongo for an entity with the id "urn:ngsi-ld:test_entity_id", the query is something like:

select * where _id.id == urn:ngsi-ld:test_entity_id

Imagine if any key (_id, id, etc) had a dot in its name.
The query wouldn't work.
It would be like trying to have a slash as part of a filename in a Unix filesystem.

So, all keys in mongo, if a dot is present, that dot mest be replaced with something else.
We picked '='.

So, that part looks good now. It was really weird for me :)
I still have no explanation for the backslashes though, and I just finished another functest trying to see the error but it all works just fine.  I suspect the backslashes come from wherever you host your context.  Not sure though, but that's what I suspect.
I'll upload the functest as a PR and I'll add a link to it hear once it's merged. I'd like you to have a look at it.

> Even in the old way with @context they still have . in attrNames. I don't know what's causing this.

That's how it should be.
It's inside the object "attrs", where the attribute longnames are keys in mongo that the dots need to be replaced.
All good.

> If you have any questions, please don't hesitate to ask. I will help find the problem in any way I can

Yeah, I just want you to take a careful look at your context where it is hosted.
Especially, the request you send when you update the hosted context, before you ask the broker to DELETE+Reload it.

[kzangeli]

Here you can see the functest as part of the PR (that I will probably merge tomorrow - it's getting late):
https://github.com/FIWARE/context.Orion-LD/pull/1719/files.

I believe the functest is pretty self-explanatory, but very short:

What's in "--SHELL-INIT--" is startup stuff. The DB is cleaned, the broker is started and stuff like that.

Then comes "--SHELL--", that's the actual commands used for the test, especially a function called orionCurl, that's just a wrapper for curl.

After that comes "--REGEXPECT--". That part is compared with what came out on stdout when the SHELL part executed.
It's an "intelligent diff" and if there's not a match, the test fails.

Last part is called "--TEARDOWN--" and I'm not even gonna explain what that is ÷)

[AizenvoltPrime]

context.txt
I attached the complete context. I copy pasted it exactly from my website page. The only difference is the URL part mysite.com where it should be the original site name. As for the DELETE+Reload I tried the process I explained to you before, meaning create entity, subscription and add data on a clean local project and it still had those problems with backslashes and duplicate attributes so I don't think it has anything to do with DELETE+Reload. I believe if you try the same requests in that I gave above with my docker compose setup you will get the same results.

[kzangeli]

ok, I could use that one instead in my functest, see if it changes anything.

However, that's one context of yours, but then you modify it and you push it again, from what I could understand.
That's one place the backslashes could come in.
Perhaps it could also be libcurl that is playing with us ... (what what the broker uses to download the contexts on a DELETE+Reload). Only, if that was the case, my functest would see the same issue.

I'll start adding traces and we'll use logging to find out more.
Very important to know whether the backslashes come in to the broker as part of the context, or if it is the broker itself that add them for some strange reason.

I'll add traces tomorrow, see if we learn anything more.

Pretty sure t he backslashes are in the @context, at least in the context cache of the broker.
We wouldn't see them otherwise ...

Mañana más (more tomorrow) :)

[AizenvoltPrime]

Ok we will talk more tomorrow. One thing I need to point out, in the local test I made on a clean docker compose with no data at all I just created an entity, subscription and then added data. I never run DELETE+Reload request there, nor did I make a change in the context from what I send you in the previous message and the same problems happened again.

[kzangeli]

What if you were to upload your context to somewhere I can see it?

[AizenvoltPrime]

I sent you an email with the link for the context I use for orion. Now you have everything I do and you can replicate exactly what I did.

[kzangeli]

Yeah, saw it. And I also saw the backslashes in there.
Responded by email.

I'm now preparing a very simple functest, just creating an entity using your context and then seeing the entity in mongo.
Should be enough.
I'll let you know as soon as it's done (lots of meetings today ... :( )

[AizenvoltPrime]

Ok now we know the problem, I guess if I remove the backslashes from the context, the other problem with duplicate entities will be fixed. Although as I mentioned in my other comment above:

I noticed and you mentioned it as well that the mongo entity of context doesnt change from the original one after running the delete request. The attributes that were added to the context after the initial load are expanded correctly after I update the context in the site page and run the delete request. The problem I have is since the mongo db entity for context doesnt change that means if orion container stops or is deleted the attributes that were added to the context after the initial download wont be included in the context anymore and if I try to create a new entity with those attributes without reloading the context again they wont have the correct url. Is there a way to force orion to recreate the context entity in mongo as well as refreshing its context cache? If there isn't I guess I have to go back to the previous way of creating entities without link because my context is dynamic since users in my app can add attributes to the context when they create a new entity that has an attribute that doesnt exist in the context.

I don't think the solution with the link is the way to go for my implementation for the above reasons. Since my context can change and there is no official way to delete the downloaded context in mongo with DELETE+Reload you gave since that only affects the container, then if the container stops or is deleted all the additional context after the downloaded was saved in mongo will be lost and I will need to manually delete and reload again.

[kzangeli]

I guess if I remove the backslashes from the context, the other problem with duplicate entities will be fixed

That would be my guess as well. Try it and let me know.

Now, I need to explain a little about the @context, as you are expecting some kind of "connection" between an entity and the context that was used in its creation.

There is no connection whatsoever.
The context is given in a request, and it is used during that request, to expand and to compact the items of the payload.
And that is all. When the request ends, the context "dies".
However, it would be very stupid of me not to save the context as it is very costly to download it and put it in my cache.
So, I invented the context cache and every context is downloaded only once. The URL of the context is used as its id.

If you then modify your context, there is no way for the broker to know that (in will stay "old" in the brokers context cache) and that's why we invented the DELETE+Reload thing.
But, that will only update the context, never any entity that was created/modified/etc using the context.

Imagine you create an entity with context A, then you update the entity 15 times with 15 different contexts.
Now we have 16 different contexts that have been used to create/modify the entity.
These 16 contexts could perfectly have different longnames for one and the same shortname.
It just wouldn't make any sense to have any connection between a context and an entity.

Also, remember that the longname is the real identity of an attribute.
The shortnames are just aliases, for it to be easier for the users, as attribute names are very long.
If you change the longname of some shortname, then you're referring to a different attribute with that same shortname.
So careful with that.

If you want this "dynamic context", I'd propose you only add items to it. Never change any items.

Ah, and a bug I noticed while looking at your issue:
When you update a context with DELETE+Reload, it is updated in the context cache but it seems like I've forgotten to persist the new updated context in mongo. I'll try to fix that asap.

If you feel you need any clarifications on all this, let's set up a meeting and I'll answer any questions as well as I can.
Just let me know. You know where to find me :))

[kzangeli]

Ah, and one more thing, important one.

It's about all those fields you have in your context, e.g.

        "temperature": {
            "@id": "myAttributes:temperature",
            "@type": "xsd:float",
            "unit": "°C",
            "en": "Temperature",
            "el": "Θερμοκρασία"
        }

I imagine it's correct JSON-LD (haven't checked), but, Orion-LD will only be using the "@id" field out of all those fields.

For various reasons I implemented my own library for handling of the context.
And, I've implemented only the part of JSON-LD that is actually used by NGSI-LD.

While it would be fairly easy to add checks for "@type", I'd prefer not to, as it affects the performance of the broker.
This type check is really part of your smart data model, and an NGSI-LD broker is agnostic to the data model. That was the first decision that was made when starting to define the API.

About "unit", we define a specific non-reified sub-attribute, called "unitCode" for that purpose.
It's part of the attribute itself, not part of the @context.

And, for different languages, that is taken care of by the context.
If you want to use greek shortnames (or any other language), that is fine and actually quite common.

You'd create a new context, with items like this;

"Θερμοκρασία": "myAttributes:temperature"

Using a context like that, you can express your entity with greek shortnames.
You actually don't need anymore than that one single line. Same same for an "english context":

  "temperature":  "myAttributes:temperature"

Well, you'd need the prefix as well, of course (myAttributes).

[AizenvoltPrime]

I think I may not have explained correctly what I was trying to do. Lets say the context that I host on my site looks like:

{
    "@context": {
        "myAttributes": "https://mysite.com/api/ngsi-ld-attributes/",
        "temperature": {
            "@id": "myAttributes:temperature",
            "@type": "xsd:float",
            "unit": "°C",
            "en": "Temperature",
            "el": "Θερμοκρασία"
        }
   }
}

I create an entity with that context:

curl --location 'http://SERVER_IP:1026/ngsi-ld/v1/entities/' \
--header 'Accept: application/json'\
--header 'Content-Type: application/json' \
--header 'NGSILD-Tenant: openiot' \
--header 'Link: <https://mysite.com/api/ngsi-ld-attributes>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
--data '{
  "id": "urn:ngsi-ld:test_entity_id",
  "type": "test_entity_type",
  "temperature": {
    "type": "Property",
    "value": 23.4
  }
}'

and at that moment since its the first time orion sees this link it downloads the context and saves it in mongo and in its cache.
Everything good till now. Next lets say a user wants to create a new entity that has a single attribute named humidity. What I would do is add the attribute to the context page on my site then make the DELETE+Reload request you gave me and then make the request to create that new entity with the attribute humidity:

curl --location 'http://SERVER_IP:1026/ngsi-ld/v1/entities/' \
--header 'Accept: application/json'\
--header 'Content-Type: application/json' \
--header 'NGSILD-Tenant: openiot' \
--header 'Link: <https://mysite.com/api/ngsi-ld-attributes>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
--data '{
  "id": "urn:ngsi-ld:test_entity_id",
  "type": "test_entity_type",
  "humidity": {
    "type": "Property",
    "value": 23.4
  }
}'

The entity will be created just fine. The problem is that even though the context in the cache of the orion container will be reloaded and the new context which is:

{
    "@context": {
        "myAttributes": "https://mysite.com/api/ngsi-ld-attributes/",
        "temperature": {
            "@id": "myAttributes:temperature",
            "@type": "xsd:float",
            "unit": "°C",
            "en": "Temperature",
            "el": "Θερμοκρασία"
        },
        "humidity": {
            "@id": "myAttributes:humidity",
            "@type": "xsd:integer",
            "unit": "%",
            "en": "Humidity",
            "el": "Υγρασία"
        }
    }
}

will be added to orion container, the MongoDB context will remain:

{
    "@context": {
        "myAttributes": "https://mysite.com/api/ngsi-ld-attributes/",
        "temperature": {
            "@id": "myAttributes:temperature",
            "@type": "xsd:float",
            "unit": "°C",
            "en": "Temperature",
            "el": "Θερμοκρασία"
        }
   }
}

so if for example the orion container stops or is deleted and I start it again the context that it will know is the one saved in Mongodb. So if I try to create an entity that has any of the attributes that didn't exist in the initial context that was saved in MongoDB like humidity, the entity saved in mongo wont have the correct URL for that attribute and in order to have it I will need to do DELETE+Reload first thing after I start the orion container. I don't plan on editing existing attributes in the context, I just want to be able to add attributes and for the MongoDB to also be in sync with the orion container so that if orion container stops and starts again, it will always have the MongoDB as backup to stay up to date with the latest context.

[kzangeli]

ok, good. That's the exact bug I discovered yesterday.
For some reason, the context isn't updated in mongo after a DELETE+Reload.
I'll try to fix this error asap. Hopefully this week, definitely before Christmas

[AizenvoltPrime]

It seems I stumbled upon quite a few things that none had stumbled upon before. I think its good since now the orion ld project will be more production-ready. I learned a lot about how Orion works with these interactions. Thanks a lot.

[kzangeli]

Yes, right back at you.
Always happy to receive issues (Free QA) and make the broker a little bit better :)

[AizenvoltPrime]

Ok removed the backslashes from URL, tested it with Link and everything works fine now. Once you fix the mongo db bug for DELETE+Reload then it will be perfect.

[AizenvoltPrime]

In regards to your input on context and how I use translations and units in there. My thinking about context besides the obvious use for the entity creation and subscription, is to be more like a reference page where if someone wants to find data on the attributes I use for any entity they can go there and fetch the data. I don't want to overcomplicate the structure of entities with adding unitCode and I don't want to have separate attributes that are the same as other attributes but in different language. I want a concise structure for the context page so even though it doesn't offer any benefit in regards to orion, it can help if for example some other service wants to use my entities for their benefit, to have a reference page of what the attributes I have used represent. It could also help if I want to add AI in the app later to have a concise structure with unique attributes rather than multiple attributes that represent the same thing. I don't know if my thinking is wrong. I just wanted to explain my thought process.

[kzangeli]

ok, that's all fine. No problem whatsoever.
Just bear in mind that the broker is ignoring most of that.

I fixed the bug, btw. during my second meeting before lunch.
Pretty sure I'll merge the PR (functest-suite is running) in an hour or two.

[AizenvoltPrime]

Just bear in mind that the broker is ignoring most of that.

Yes thanks to you I am aware of that, before I thought it made a difference but in the end it doesn't really matter to me if it ignores the rest of the info.

I fixed the bug, btw. during my second meeting before lunch.
Pretty sure I'll merge the PR (functest-suite is running) in an hour or two.

Oh that was faster than I thought. Good job!

[kzangeli]

Boring meeting and easy fix. It's the perfect combination to get something done :)))

[kzangeli]

OK, PR merged and everything deployed to dockerhub and quay.io.
My very simple functest is working, the new context is now pushed to mongo.
So, try it and let me know

[AizenvoltPrime]

Tested it and it works perfectly now with link and Delete+Reload. Should I close the issue or wait for you to fix the problem I originally had when I created this issue?

[kzangeli]

Yeah, I think it is all taken care of now :)

Thank you so much for finding these issues. It's been good!
Wait, what was that "original" thingy?

[AizenvoltPrime]

So, I found the error. I explain:
As you are creating the subscription with a compound @context, the broker needs to create and host this context.
This new context is hosted and added to the cache and all is working just fine.
However, it seems I have forgotten to persist the new context in the database, so when the broker is restarted, the context is gone!!!
Important bug of mine. I feel a little bit ashamed to tell you the truth.
As the context is done, your attribute "temperature" will no longer be expanded the same way and thus, notifications will not get triggered - QL will receive nothing after the restart.

So, if you follow my advice and stop using "application/ld+json", the complication with the complex @context disappears and your problem is gone.

Host your context somewhere (Orion-LD offers that service, but most people use github or some web-server)
Use the Link header to pass the context to the broker, and always "application/json", both for Content-Type and Accept.
I will of course proceed and fix my bug

[kzangeli]

ok, so, yeah, that is fixed, in this PR: #1718.

[AizenvoltPrime]

I tested it just now locally to make sure and it still doesn't work correctly.

[kzangeli]

Really?
ok, then I will need to take a look.