Evaluation in Wolfram System Modeler

[beutlich]

@otronarp @maltelenz I wonder how the library v3.1.0 evaluates in WSM. SinceI do not have access to WSM it would be helpful if you could run the examples and give feedback here. Thanks.

[maltelenz]

@beutlich I tested it with System Modeler on Linux-64.

These seem to work fine:

CSVTest
MATTest
TIRTest

For many examples I get this from the simulation:

ModelicaError: Syntax error at character 1 of
                 
                 *
                 Expected a Real number 

These are the examples with said error message:

INITest
JSONTest
XSLTest
XSLXTest
XMLTest
XMLTestXPath

For SSVTest I get an assert during simulation:

Error: [ASSERT] [?:?] In ExternData.Functions.SSV.getRealArray2D: SSV version needs to be greater than or equal to 2.0 [@0.0 s]

For XMLTestInnerOuter I get a build failure. I suspect this may be a System Modeler bug:

Error: ExternData.Examples.XMLTestInnerOuter.Component [8:51-8:51] Element not found. When looking up dataSource.getRealArray2D in scope ExternData.Examples.XMLTestInnerOuter.Component, getRealArray2D was not found in ExternData.Interfaces.DataSource.
Error: ExternData.Examples.XMLTestInnerOuter.Component [7:7-7:7] Element not found. When looking up dataSource.getArrayRows2D in scope ExternData.Examples.XMLTestInnerOuter.Component, getArrayRows2D was not found in ExternData.Interfaces.DataSource.
Error: ExternData.Examples.XMLTestInnerOuter.Component [5:39-5:39] Element not found. When looking up dataSource.getString in scope ExternData.Examples.XMLTestInnerOuter.Component, getString was not found in ExternData.Interfaces.DataSource.
Error: ExternData.Examples.XMLTestInnerOuter.Component [4:39-4:39] Element not found. When looking up dataSource.getReal in scope ExternData.Examples.XMLTestInnerOuter.Component, getReal was not found in ExternData.Interfaces.DataSource.

For XMLTestReadDim, we don't have support for evaluating the external code during build yet (this is a feature we are working on).

[maltelenz]

(I'm guessing the library is still illegal Modelica per modelica/ModelicaSpecification#2399 , so I'm disappointed we don't just reject all examples during build.)

[beutlich]

Thanks for evaluating.

These seem to work fine:

Did you also check the results?

For many examples I get this from the simulation:

These models call Modelica.Utilities.Strings.scanReal which fails on empty string.

For SSVTest I get an assert during simulation:

I belive it hides the same issue of "empty" values.

For XMLTestReadDim, we don't have support for evaluating the external code during build yet (this is a feature we are working on).

That model actually is a workaround only.

I'm guessing the library is still illegal Modelica

Hopefully it is valid Modelica though there are spec. issues on external objects that should have been clarified years ago.

[maltelenz]

Did you also check the results?

I plotted them, but just to check they contained data. I didn't verify against anything.

Hopefully it is valid Modelica though there are spec. issues on external objects that should have been clarified years ago.

https://specification.modelica.org/master/class-predefined-types-and-declarations.html#S7.I1.i1.p1 says:

The components directly declared in a record may only be of specialized class record or type.

which seems to exclude ExternalObject pretty clearly.

[beutlich]

If it's just the records - they got introduced by #18. Here's the same lib with that change reverted back to model: https://github.com/modelica-3rdparty/ExternData/tree/no-records Does it work in WSM?

[maltelenz]

With the no-records branch, only the problems with XMLTestInnerOuter (I suspect a System Modeler issue) and XMLTestReadDim (definitely a System Modeler missing feature) remain.

The results all look like this:

Except for TIRTest which looks like this:

[maltelenz]

After further reading of the specification, it is my interpretation that WSM does the right thing for XMLTestInnerOuter, when it reports:

Error: ExternData.Examples.XMLTestInnerOuter.Component [8:51-8:51] Element not found. When looking up dataSource.getRealArray2D in scope ExternData.Examples.XMLTestInnerOuter.Component, getRealArray2D was not found in ExternData.Interfaces.DataSource.
Error: ExternData.Examples.XMLTestInnerOuter.Component [7:7-7:7] Element not found. When looking up dataSource.getArrayRows2D in scope ExternData.Examples.XMLTestInnerOuter.Component, getArrayRows2D was not found in ExternData.Interfaces.DataSource.
Error: ExternData.Examples.XMLTestInnerOuter.Component [5:39-5:39] Element not found. When looking up dataSource.getString in scope ExternData.Examples.XMLTestInnerOuter.Component, getString was not found in ExternData.Interfaces.DataSource.
Error: ExternData.Examples.XMLTestInnerOuter.Component [4:39-4:39] Element not found. When looking up dataSource.getReal in scope ExternData.Examples.XMLTestInnerOuter.Component, getReal was not found in ExternData.Interfaces.DataSource.

The functions that XMLTestInnerOuter.Component tries to access through dataSource, do not exist in the outer ExternData.Interfaces.DataSource dataSource type in ExternData.Interfaces.DataSourceModel, so they cannot be accessed through the outer component.

This is the relevant part of the specification:

The inner component shall be a subtype of the corresponding outer component.

That part is fine.

[If the two types are not identical, the type of the inner component defines the instance and the outer component references just part of the inner component.]

This example is trying to reference parts from the inner component, but only the parts from the outer component are available.

[beutlich]

Thanks for the feedback.

Regarding the XMLTestInnerOuter example. Does it help if the functions are added to ExternData.Interfaces.DataSource like in https://github.com/modelica-3rdparty/ExternData/tree/fix-inner-outer?

Otherwise, I believe I can safely remove this example and the corresponding types ExternData.Interfaces.DataSource, ExternData.Interfaces.DataSourceModel and ExternData.Interfaces.DataSourceBlock.

[maltelenz]

Thanks for spending time on this.

The fix-inner-outer makes the example work in WSM.

I don't know what the spec says (and our local expert is unavailable at the moment), but to me it seems a little bit sketchy that the new partial functions don't have the inputs they are called with in ExternData.Examples.XMLTestInnerOuter.Component. Maybe this is fine, at least WSM isn't complaining about it.

[beutlich]

OK, so it seems WSM is another tool to be added as supported (in the Readme). I ported the fix-inner-outer changes to master and rebased the no-records branch accordingly.

My other question is, if it is worth to keep the no-records branch alive, maintain (and release) it as alternative version.

[maltelenz]

My other question is, if it is worth to keep the no-records branch alive, maintain (and release) it as alternative version.

I would be curious to see the warning from Dymola mentioned in #17, to see if there is something else illegal with the non-record variant WSM should have caught.

[beutlich]

You might have noticed that I needed to remove the ExternData.Examples.XLSTest.computeColSum and ExternData.Examples.XLSXTest.computeColSum functions from the no-records branch because a specialized class model should not be passed as input argument to a function.

The other advantage of records over models is that it can be declared as parameter.

[maltelenz]

Thank you, I hadn't noticed that.

It looks like only dataSource.xls is used in computeColSum, and that looks like an ExternalObject, which should be fine to send to functions? Maybe then computeColSum could be expressed legally like this?

function computeColSum "Compute column sum"
  extends Modelica.Icons.Function;
  input ExternData.Types.ExternXLSFile xls "Excel XLS file record";
...
  input Integer endRow = ExternData.Functions.XLS.getArrayRows2D(sheetName=sheetName, xls=xls) "End row";
...
end computeColSum;

and called like this?

final parameter Real sumB = computeColSum(dataSource.xls, "B")

Unfortunately I'm on my mac today, so I can't test it past validation in WSM (since ExternData doesn't have macOS binaries).

[beutlich]

Yes, that is possible and certainly a reasonable change. Fixed by 713ede3 in master. The branch no-records is rebased accordingly.

Regarding the missing macOS binaries: Would be some work to add the hdf5 lib specifically.

[maltelenz]

Then the remaining question is whether being able to declare the dataSource a parameter is worth the library being illegal Modelica?

[ankitnaik177]

I tested the no-records branch on Windows using System Modeler, and none of the examples build.

I get the following error:

`sme.14.3.0_1738155665_1533310374_ext1-get.obj : error LNK2005: timezone already defined in sme.14.3.0_1738155665_1533310374_ext0-get.obj

sme.14.3.0_1738155665_1533310374_ext1-get.obj : error LNK2005: _arrayTimezone_initialize already defined in sme.14.3.0_1738155665_1533310374_ext0-get.obj

sme.14.3.0_1738155665_1533310374_ext1-get.obj : error LNK2005: Timezone_initialize_wrapper already defined in sme.14.3.0_1738155665_1533310374_ext0-get.obj

sme.14.3.0_1738155665_1533310374_ext1-get.obj : error LNK2005: __iob_func already defined in sme.14.3.0_1738155665_1533310374_ext0-get.obj`

[maltelenz]

As @ankitnaik177 mentioned, we do get linking errors on Windows. I think this is because we build each external function separately, and then link everything together.

Maybe something like __declspec(selectany) is needed (from https://learn.microsoft.com/en-us/cpp/error-messages/tool-errors/linker-tools-error-lnk2005?view=msvc-170 ) if definitions are required to be in the header file msvc_compatibility.h?

[beutlich]

Following the TDD approach and easiest for me is to have the linker error reproducable in the CI job test_cmake_windows-msvc. Feel free to file a PR. Otherwise it is too much guessing for me.

[beutlich]

Then the remaining question is whether being able to declare the dataSource a parameter is worth the library being illegal Modelica?

Since the library has several users on Dymola I need to consider compatibility and must not break existing models if #18 gets reverted. Hence I hope for a clarification in Modelica Language first and rather would postpone the decision until modelica/ModelicaSpecification#2399 is resolved.

[beutlich]

As @ankitnaik177 mentioned, we do get linking errors on Windows. I think this is because we build each external function separately, and then link everything together.

I could easily reproduce in the unit test setup. I added ee72766 to the no-record branch that demonstrates both the issue and its mitigations. Using the /FORCE:MULTIPLE linker otption on your build commands will still be required, for the price of unavoidable linker warnings, see https://github.com/modelica-3rdparty/ExternData/actions/runs/13039655564/job/36378400320#step:5:1

[maltelenz]

Using the /FORCE:MULTIPLE linker otption on your build commands will still be required, for the price of unavoidable linker warnings

I'm hesitant to add that flag to System Modeler.

The specification puts this requirement squarely on the library side:

The Include annotation shall be used in such a way that each external function can be handled in a separate translation unit. In particular, different external functions must not have Include annotations providing exported definitions of the same function symbol to avoid linking errors.

maltelenz@90168c1 seems to avoid two more cases, but I wasn't able to get rid of this final one:

sme.14.2.0_1738224744_2466028096_ext1-get.obj : error LNK2005: __iob_func already defined in sme.14.2.0_1738224744_2466028096_ext0-get.obj

[beutlich]

but I wasn't able to get rid of this final one:

Did you try to set the linker option /FORCE or /FORCE:MULTIPLE? This should resolve the issue.

I'm hesitant to add that flag to System Modeler.

SimulationX has set /FORCE for years, in case you need some other example.

The specification puts this requirement squarely on the library side.

Ah, never noticed this before. Thanks for the update. And I now see where the new paragraphs originate from: modelica/ModelicaSpecification@7fe3b48 for modelica/ModelicaSpecification#3452 / modelica/ModelicaSpecification#3451. In that sense, ExternData is still on MLS 3.6 (being latest officially MA approved MLS): https://specification.modelica.org/maint/3.6/functions.html#annotations-for-external-libraries-and-include-files and as such somehow relys on features that might get deprecated in a future MLS version. So it's kind of a non-issue, right?

Modelica-wise, how can a library developer guarantee that a certain "Include" content is built exactly once - under all circumstances - as compilation unit? Inner/outer following the Modelica.Mechanics.MultiBody.World concept might a way to go. Let me know if I should give it a try? Well, I guess I should do it.

[beutlich]

The entire MSVC compatibility hack can be avoided if all libraries are compiled by the same compiler as the final binary. That's the proper way to go.

[maltelenz]

Did you try to set the linker option /FORCE or /FORCE:MULTIPLE? This should resolve the issue.

No, I didn't try that yet. I am still hoping to avoid it.

In that sense, ExternData is still on MLS 3.6 (being latest officially MA approved MLS): https://specification.modelica.org/maint/3.6/functions.html#annotations-for-external-libraries-and-include-files and as such somehow relys on features that might get deprecated in a future MLS version. So it's kind of a non-issue, right?

Technically, this is correct. In practice, WSM already builds external functions in separate units, since putting everything in the same translation unit/file has other, harder to fix, problems.

Modelica-wise, how can a library developer guarantee that a certain "Include" content is built exactly once - under all circumstances - as compilation unit?

This is not what the change to the specification asks for. What the specification asks for is that one/each external function can be built standalone using its include annotation, in a way that many such builds (of different functions) can later be linked together into one binary. This is to allow tools to build each external function separately, so that different external functions do not interfere with each other (polluting namespace using include, defines, ...).