Reduce tolerance by a factor 10 when generating reference trajectories for testing

[casella]

While analyzing OMC's verification failure reports with @AndreaBartolini, we stumbled upon one case which made us question the whole thing. Consider the Buildings.Airflow.Multizone.Examples.OneOpenDoor.diff.dooOpeClo.port_a2.m_flow: the reference trajectory

shows a glitch at time = 2418 which is clearly non-physical, but rather a numerical simulation artefact. We understand the reference trajectories were generated with Dymola 2024x Refresh 1, but we don't know what tolerance was used for that.

If I use Dymola 2024x and Dymola 2024x Refresh 1 with the tolerance as specified by the experiment annotation (1e-6) we get some visible glitches, but not as large as the one shown above. If we use a tighter tolerance 1e-7, we get a very smooth trajectory.

For the MSL, the standard accepted practice is to generate the reference trajectories with a tolerance reduced by a factor 10, to make sure that the numerical artefacts of the reference trajectory (if any) are much smaller than the ones obtained during simulation. I think you should do the same also when generating reference data for the Buildings library. It is likely that doing so will solve many false negatives in the verification process against OMC.

@mwetter can you comment on what is the current status and if this solution is OK for you?

[mwetter]

@casella : For generating the reference trajectory, we use the tolerance as stated in the .mo file. Globally changing the tolerance is not a good idea in my opinion as not all models work if the tolerance is changed. This would make it hard to ensure that models simulate, or simulate within the stated timeout, with the tolerance of the library, and with a 10x reduced tolerance. I therefore suggest to increase the tolerance only on the models where we have such numerical glitches.

[casella]

@mwetter thanks for the feedback!

The problem with this approach could be that the time it takes to generate a really good reference trajectory is, say, 10X larger than the time it takes to get them with a few glitches. Would you sacrifice the user experience when simulating those models with the experiment annotation?

To me, running "decent" simulation and running "reference" simulations are two different tasks.

My take is that reducing by 10 the tolerance to generate the reference trajectories should work in most cases (as it does e.g. for MSL), and those few where it doesn't should be handled by special annotations.

Anyway, I would warmly invite you to participate to the discussion in modelica/ModelicaSpecification#3473, where we are debating these topics, trying to established agreed-upon procedures and language support for them. We're still far from the goal, but I think your input would be valuable, given your extensive experience with non-trivial models.