Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix resend historic data function #2946

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from
Open

Fix resend historic data function #2946

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion ...nems.backend.timedata.influx/src/io/openems/backend/timedata/influx/TimedataInfluxDb.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,18 @@

@Override
public void write(String edgeId, ResendDataNotification data) {
// TODO Auto-generated method stub
if (this.config.isReadOnly()) {
return;

Check warning on line 163 in io.openems.backend.timedata.influx/src/io/openems/backend/timedata/influx/TimedataInfluxDb.java

View check run for this annotation

Codecov / codecov/patch

io.openems.backend.timedata.influx/src/io/openems/backend/timedata/influx/TimedataInfluxDb.java#L163 

Added line #L163 was not covered by tests
}

// Write data to default location
this.writeData(//

Check warning on line 167 in io.openems.backend.timedata.influx/src/io/openems/backend/timedata/influx/TimedataInfluxDb.java

View check run for this annotation

Codecov / codecov/patch

io.openems.backend.timedata.influx/src/io/openems/backend/timedata/influx/TimedataInfluxDb.java#L167 

Added line #L167 was not covered by tests
edgeId, //
data, //
(influxEdgeId, channel) -> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are actually 2 reasons why we didnt implemented this one yet.

  1. (minor reason) If you have lots of edges online which are sending lots of data and your database is already overflowing with data you may not want to have extra load on your database

  2. Existing data could be overwritten with different data. If local data and remote (backend) data is not stored equally resending data could result in different data than before.
    Local Rrd4j stores data aggregated to every 5min with the timestamp of the starting timerange (data from 13:40 to< 13:45 would have the timestamp of 13:40)
    For cumulated values (energy) that would mean that if we lose connection lets say at 13:43 we have data until then and after resending the data our resend value at 13:40 could be greater than the values after that one because its actually the maximum value of 13:40-13:45. Then we could add 4min and 59seconds to make sure this value is at the end...
    For average values (power) after resending there could also be a data change because we add one more value to a timerange and therefore the average of that timerange could be different.

You can definitly add this but you should definitely have this in mind while using it.
I think the best would be to add an config option to enable/disable it with a doc with the disadvantages of it.

Also if you havent seen it thats why we actually use the AggregatedInflux it doesnt have those flaws but only stores defined data

this.timestampedChannelsForEdge.put(influxEdgeId, channel);
return true;

Check warning on line 172 in io.openems.backend.timedata.influx/src/io/openems/backend/timedata/influx/TimedataInfluxDb.java

View check run for this annotation

Codecov / codecov/patch

io.openems.backend.timedata.influx/src/io/openems/backend/timedata/influx/TimedataInfluxDb.java#L171-L172 

Added lines #L171 - L172 were not covered by tests
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just return true and remove this.timestampedChannelsForEdge.put(influxEdgeId, channel); this is only relevant for "live" data

});
}

private boolean isTimestampedChannel(int edgeId, String channel) {
Expand Down
7 changes: 6 additions & 1 deletion io.openems.edge.timedata.rrd4j/src/io/openems/edge/timedata/rrd4j/Rrd4jReadHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,7 @@
}
continue;
}
var channelType = channel.getType();

Check warning on line 216 in io.openems.edge.timedata.rrd4j/src/io/openems/edge/timedata/rrd4j/Rrd4jReadHandler.java

View check run for this annotation

Codecov / codecov/patch

io.openems.edge.timedata.rrd4j/src/io/openems/edge/timedata/rrd4j/Rrd4jReadHandler.java#L216 

Added line #L216 was not covered by tests
try (final var database = this.rrd4jSupplier.getExistingUpdatedRrdDb(//
rrdDbId, channel.address(), channel.channelDoc().getUnit())) {
if (database == null) {
Expand Down Expand Up @@ -255,9 +256,13 @@
continue;
}

var jval = switch (channelType) {
case FLOAT, DOUBLE -> new JsonPrimitive((float)value);
default -> new JsonPrimitive((int)value);

Check warning on line 261 in io.openems.edge.timedata.rrd4j/src/io/openems/edge/timedata/rrd4j/Rrd4jReadHandler.java

View check run for this annotation

Codecov / codecov/patch

io.openems.edge.timedata.rrd4j/src/io/openems/edge/timedata/rrd4j/Rrd4jReadHandler.java#L260-L261 

Added lines #L260 - L261 were not covered by tests
};
// return timestamps in milliseconds
resultMap.computeIfAbsent(timestamp * 1000, t -> new TreeMap<>()) //
.put(channelAddress, new JsonPrimitive(value));
.put(channelAddress, jval);

Check warning on line 265 in io.openems.edge.timedata.rrd4j/src/io/openems/edge/timedata/rrd4j/Rrd4jReadHandler.java

View check run for this annotation

Codecov / codecov/patch

io.openems.edge.timedata.rrd4j/src/io/openems/edge/timedata/rrd4j/Rrd4jReadHandler.java#L265 

Added line #L265 was not covered by tests
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is just for reading resend data. Shouldnt this be also applied to e. g. local history request as well?
Maybe we should already save the values "correctly" = rounded?

Basically same as for sending to the backend:

openems/io.openems.edge.controller.api.backend/src/io/openems/edge/controller/api/backend/SendChannelValuesWorker.java

Lines 206 to 219 in 5eae9c5

var value = channel.getPastValues() //
.tailMap(channelStartTime, true) //
.entrySet() //
.stream() //
.filter(e -> e.getKey().isBefore(endTime.toLocalDateTime())) //
.filter(e -> e.getValue().isDefined()).map(e -> e.getValue().get()) //
.collect(aggregateCollector(channel.channelDoc().getUnit().isCumulated(), //
channel.getType()));
// TODO aggregation should be modifiable in Doc e. g. not every EnumDoc may want
// this behaviour
if (channel.channelDoc() instanceof EnumDoc) {
value = aggregateEnumChannel(channel, channelStartTime, endTime.toLocalDateTime());
}

}
}

Expand Down
Loading