Error in JSON encoding

[larjohn]

I have received the following bug report recently:

http://apps.openbudgets.eu/cube/analytics/bonn-budget-2022-PSPelement__2603c/outlier_detection/LOF_FACTS?BABBAGE_FACT_URI=http%3A%2F%2Fapps.openbudgets.eu%2Fapi%2F3%2Fcubes%2Fbonn-budget-2022-PSPelement__2603c%2Ffacts%3F&coloringAttribute=fiscalYear.prefLabel&groupingAttribute=profitCenter.prefLabel
(return Error: 0)

http://apps.openbudgets.eu/cube/analytics/bonn-budget-2022-PSPelement__2603c/descriptive_statistics/facts?amounts=amount&dimensions=operationCharacter.prefLabel%7CprofitCenter.prefLabel%7CbusinessArea.prefLabel%7CeconomicClassification.prefLabel%7Corganization.organization&json_data=http%3A%2F%2Fapps.openbudgets.eu%2Fapi%2F3%2Fcubes%2Fbonn-budget-2022-PSPelement__2603c%2Ffacts%3F
(Error: lexical error: invalid char in json text. und Versicherungen","amount":NAN},{"operationCharacter.prefL (right here) ------^ In call: parse_con(txt, bigint_as_char))

It is caused by a NAN (not a number) being literally dumped as-is by PHP's json_encode function. NAN is a valid structure in PHP, but it has no valid meaning in a JSON serialization.

PHP docs state that there is a constant to provide as an option so that INF (infinity) and NAN (not a number) are substituted with zeros during the serialization. I tried the following chunk of code:

<?php
echo json_encode(["a" => INF, "b"=> NAN],JSON_ERROR_INF_OR_NAN |  JSON_PARTIAL_OUTPUT_ON_ERROR | JSON_PRETTY_PRINT );
?>

In my local Windows PHP 7.0 installation, the above correctly resulted in {"a":0, "b":0}. In our docker image, INF was correctly converted to 0, but NAN remained as-is. I tried manually updating the php7 version in the running instance to PHP 7.1 and it worked correctly. It seems it was a bug and it got fixed.

While I could update PHP myself, this might result to some error driving the server out of order. So, @wk0206 can you or someone else just update the Rudolf docker image to a newer (7.1) version?

[larjohn]

@HimmelStein (i think it was your bug report) can you check why this observation:
DESCRIBE <http://data.openbudgets.eu/resource/dataset/bonn-budget-2022-PSPelement/observation/10>
has a NAN amount value, that causes the bug above?

[fathoni]

Hi @larjohn , in the dataset, there is no NaN value for the mentioned observation. In the dump turtle file, the observation looks like this:

<http://data.openbudgets.eu/resource/dataset/bonn-budget-2022-PSPelement/observation/10> a qb:Observation ;
	bonn-dimension:operationCharacter <http://data.openbudgets.eu/resource/codelist/bonn-operation-character/expenditure> ;
	bonn-dimension:administrativeClassification <http://data.openbudgets.eu/resource/codelist/aemterhierarchie_bonn/3000> ;
	bonn-dimension:functionalClassification <http://data.openbudgets.eu/resource/codelist/produktuebersicht_bonn/0127> ;
	bonn-dimension:economicClassification <http://data.openbudgets.eu/resource/codelist/kostenartenuebersicht_bonn/529100> ;
	bonn-dimension:profitCenter <http://data.openbudgets.eu/resource/codelist/profitcenter_bonn/130000127> ;
	bonn-dimension:PSPelement <http://data.openbudgets.eu/resource/codelist/psp-elemente_bonn/1.30.00.01.27.01> ;
	bonn-dimension:businessArea <http://data.openbudgets.eu/resource/codelist/geschaeftsbereich_bonn/3099> ;
	obeu-measure:amount "0.0"^^xsd:float ;
	qb:dataSet <http://data.openbudgets.eu/resource/dataset/bonn-budget-2022-PSPelement> .

[HimmelStein]

@larjohn I checked the raw csv file, as well as the turtle file. Can you check whether the following format causes the NaN value?
obeu-measure:amount "-2.7699888E7"^^xsd:float ;

[liyakun]

@larjohn @HimmelStein updated Rudolf docker image to a newer (7.1) version, the following link works.

http://apps.openbudgets.eu/cube/analytics/bonn-budget-2022-PSPelement__2603c/descriptive_statistics/facts?amounts=amount&dimensions=operationCharacter.prefLabel%7CprofitCenter.prefLabel%7CbusinessArea.prefLabel%7CeconomicClassification.prefLabel%7Corganization.organization&json_data=http%3A%2F%2Fapps.openbudgets.eu%2Fapi%2F3%2Fcubes%2Fbonn-budget-2022-PSPelement__2603c%2Ffacts%3F

[HimmelStein]

@larjohn Indigo seems not working properly http://apps.openbudgets.eu -- Slow and sometimes datasets or data-mining functions do not prompt out . Can you have a check?

for example http://apps.openbudgets.eu/cube/analytics/bonn-budget-2022-internalOrder__f31cf

http://apps.openbudgets.eu works, if we first visit
http://apps.openbudgets.eu/search/package?q=%22%22&size=100&from=0

[larjohn]

@HimmelStein you have included conditions inside the configurations node, whereas it should be included as single condition per related configuration. Right now conditions is considered as another configuration and it lacks inputs, that's why it fails.

[HimmelStein]

@larjohn @liyakun @wk0206
Please type 'Bonn' at 'Find a Dataset' field of http://apps.openbudgets.eu/ , nothing happens
and type 'Aragon' at 'Find a Dataset' field of http://apps.openbudgets.eu/, aragon datasets will be listed. Select one aragon dataset, and click 'Analyse', the data-mining functions are not listed.

what is the efficient way to fix these problem?

[liyakun]

I don't have this problem for dataset search.

[liyakun]

the analysis problem happens to me as well

[HimmelStein]

@liyakun I tested with firefox and chrome. Response of firefox is slower than of chrome.
data-mining tools are not listed (the same as your case).

[HimmelStein]

typing 'Bonn' at 'Find a Dataset' field of http://apps.openbudgets.eu/, it takes much longer time to get response than by typing 'Aragon', although there are only 4 Bonn datasets.

[liyakun]

@HimmelStein @larjohn I also use Firefox. It seems the analysis already load everything without problems.

[HimmelStein]

@larjohn can you have a quick look where the problem is?

[skarampatakis]

@larjohn please look at this ASAP.

[larjohn]

I cannot reproduce the dataset search issue. On the other hand, the analysis algorithms should be visible if you resolve this.

Which means you have to move the conditions node from the configurations node of the timeseries algorithm into the aggregate configuration of the timeseries algorithm (it should be a child node of aggregate, not a sibling).

I am not doing it myself to avoid conflict, in case you are editing it per my comments in the DAM issue here.

[skarampatakis]

@larjohn could you elaborate more on #36 (comment) as it seems there is some kind of confusion.

[HimmelStein]

@larjohn @liyakun @hantongliu @wk0206 I tried with Chrome, selecting and loading dataset is fine, and also fast. But, data-mining functions are still not listed.

[larjohn]

This:

[wk0206]

updated dam.json for last comment.
commit

[liyakun]

@HimmelStein @larjohn @wk0206 @skarampatakis after updating dam to last commit, it works fine now.

[HimmelStein]

please do more testing. My side, it works once. does the following link work on your side?
http://apps.openbudgets.eu/cube/analytics/bonn-budget-2022-internalOrder__f31cf/outlier_detection/LOF_FACTS?BABBAGE_FACT_URI=http%3A%2F%2Fapps.openbudgets.eu%2Fapi%2F3%2Fcubes%2Fbonn-budget-2022-internalOrder__f31cf%2Ffacts%3F&coloringAttribute=profitCenter.prefLabel&groupingAttribute=economicClassification.prefLabel

[skarampatakis]

@HimmelStein It seems to be working from here. Perhaps a caching problem? Please clear your browser cache

[HimmelStein]

yes, I got this graphic for the first time. can you do it again?

[skarampatakis]

You are right, after a reload I get the following error on my console

[larjohn]

Most probably because this response looks like a weird matrsoka doll:

[HimmelStein]

@larjohn please check where is the problem when loading datasets through http://apps.openbudgets.eu/