Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing/delayed priceinfo from entso-e #29

Open
joe-ave opened this issue Feb 16, 2023 · 20 comments
Open

Missing/delayed priceinfo from entso-e #29

joe-ave opened this issue Feb 16, 2023 · 20 comments
Labels
enhancement New feature or request

Comments

@joe-ave
Copy link

joe-ave commented Feb 16, 2023

Hello Olli,

there seems to be missing/delayed priceinfo from the entso-e platform, probably something to do with this: https://transparency.entsoe.eu/news/widget?id=63ee08ae5092e77e4d3d7bf8

Could there be an idéa to have the possibility to manually enter the prices when this happen, since they exists in Nordpool?

//Jonas

@Olli69 Olli69 added the enhancement New feature or request label Feb 16, 2023
@stikstone
Copy link

Is there now problem with updating prices at the ensoe or my side. Prices for tomorrow have not been updated. Tried restarting arska node. Did not help.

@joe-ave
Copy link
Author

joe-ave commented Feb 16, 2023

Is there now problem with updating prices at the ensoe or my side. Prices for tomorrow have not been updated. Tried restarting arska node. Did not help.

It looks like problem at the entso-e platform.

//Jonas

@Olli69
Copy link
Contributor

Olli69 commented Feb 16, 2023

Hi Jonas,

Status update while writing this: Now Entso-E seems to have update prices for tomorrow.

Oh no - but thanks for letting me know about the delay and idea! That is something which is would be technically possible, but I think other, easier.., workarounds should considered first. A few points:

  • First, we should be able trust that Entso-E, the association for the cooperation of the European transmission system operators (TSOs), is able to provide up-to-date data on it's API. Last problems - if I remember right - have been fixed anyway same day. But it has been annoying to wait it...
  • Making a backup service from where Arska could fetch (manually entered) price info would be against Entso-E terms & conditions (for free use). So every user should enter prices manually - and know when there is a problem. We try to keep Arska simple to users, so entering prices manually cannot be normal procedure.
  • There could errors also in internet connection, hardware etc, so in critical applications rules and wirings should be error-prone.
  • So, let's hope that Entso-E get the problems solved, but keep your idea of manual update also in the mind.
    Olli

@joe-ave
Copy link
Author

joe-ave commented Feb 16, 2023

Hello Olli,

Yes we should be able to trust Entso-e as reliable provider, and it`s probably not very often they have this type of issues that they anyway solve the same day.

The stuff that I use Arska for works allso without Arska, only less cost efficiant. So in that meaning it’s failsafe.

So manually entering priceinfo is not needed very often, but more of a nice to have feature sometime in the future.

//Jonas

@joe-ave
Copy link
Author

joe-ave commented May 29, 2023

Hello Olli,

since the 28/5 when the pirceinfo from entso-e was totaly missing for that day the Arska-nodes does not seem to handle the new priceinfo for 29/5 in a correct way.
For the older .93 version it pasted in the values in graph from 29/5 at the place för 28/5 but the variable readout this morning looked correct.
For the newer .99 alfa 3 version ( latest from download ) it did not seemed to work at all, tried restart, power on/off and reset this morning ( 29/5 when there was ok pricedata) but variables still seemed wrong and nothing on the graph.
I´ll reflash the .99 version later today.

//Jonas

@stikstone
Copy link

stikstone commented May 29, 2023

Yep, At least .99beta1 and beta2 have this same problem. No price info since of yesterday, and flashing did not seem to help.
image

@Olli69
Copy link
Contributor

Olli69 commented May 29, 2023

Hi Jonas,
It seems that ENTSO-E is totally missing prices for Sunday 28.5.2023 and thats why Arska price query crashes on it. It is not currently build for situations that whole day is missing from Entso-E time-series (That should not happen from Entso-E's side...) and making quick fixes could break the logic. I have a look anyway. image

@Olli69
Copy link
Contributor

Olli69 commented May 29, 2023

Hi Jonas,
In addition to Entso-E time series problem there was a typo in the latest development version alfa3. Now this is fixed in alfa4 and getting SE3 prices seem to work - very quick testing. It still gives error messages and no diagram, because of missing data on Entso-E side but current price should be available in the variables.

Olli

@joe-ave
Copy link
Author

joe-ave commented May 29, 2023

Hello Olli,

thanks!, I´ll try it out later today.

//Jonas

@stikstone
Copy link

.99 alpha4 worked for me, and i got the price info back, everything working now.

@joe-ave
Copy link
Author

joe-ave commented May 29, 2023

Now it works again!

//Jonas

@joe-ave
Copy link
Author

joe-ave commented Aug 4, 2023

Hello,

there seems to be something wrong with the pricefile today from Entso-e, the price seems to be there when I look at he serial monitor but the time is probably wrong ( dump from some of the lines ):

Combined line:22
Combined line:<price.amount>18.03</price.amount>
***Invalid timeSeries index -565974
Combined line:23
Combined line:<price.amount>17.73</price.amount>
***Invalid timeSeries index -565974
Combined line:24
Combined line:<price.amount>17.14</price.amount>
***Invalid timeSeries index -565974
Combined line:

//Jonas

@joe-ave
Copy link
Author

joe-ave commented Sep 28, 2023

Hello Olli,

just want to document that priceinfo from entso-e for the Nordic countries is missing since yesterday for today 2023-09-28, it´s missing at their webpage also so not just a query thing.

//Jonas

@Olli69
Copy link
Contributor

Olli69 commented Sep 28, 2023

Hej Jonas,
Thanks for recording! Yes, now Entso-E publish announcement about that. Elering seemed to publish prices in time - unfortunately Scandinavian price areas are not included,.

Olli

@joe-ave
Copy link
Author

joe-ave commented Nov 15, 2023

Hello Olli,

just documenting that hour 35 tomorrow ( 11:00 ) has a pricevalue of 0 for Sweden and Norway when I look at the serial printout.
It should be in the area of about 11.7 cent/kw.
Denmark and Finland looks ok.
Maybee they will correct tomorrow.

//Jonas

@Olli69
Copy link
Contributor

Olli69 commented Nov 16, 2023

Hi Jonas @joe-ave ,
Thanks for the note! It corrected the value later on automatically but I catched the error and fixed it. If you have your own build and try to test it, you could replace functions getElementValue and read_http11_linewith a new fixed versions below. These need to be tested more before wider release.

Olli

Edited 16.11.23 20:07 : function read_http11_line tuned

String getElementValue(String outerXML)
{
  int s,e;
  s = outerXML.indexOf(">", 0)+1;
  e = outerXML.indexOf("</");
  if (e ==-1) // end tag missing, try to read to the end
    e = outerXML.length();
  if (s>e) // no start tag found in the beginning (but the end), get from the start of the string
    s = 0;
     return outerXML.substring(s, e);
}
String read_http11_line(WiFiClientSecure *client_https)
{
  String line;
  String line2;
  if (client_https->available())
  {
    line = client_https->readStringUntil('\n');
    if (line.charAt(line.length() - 1) == 13)
    {
      line = line.substring(0, line.length() - 1); // remove cr 
      if (client_https->available())
      {
        line2 = client_https->readStringUntil('\n');
        if (is_chunksize_line(line2))
        { // skip this line and read  next one
          Serial.print("Skipped line:");
          Serial.println(line2);
          if (client_https->available())
          {
            line2 = client_https->readStringUntil('\n');
          }
          else
            line2 = "";
        }
        return line  + line2;
      }
      else
        return line;
    }
    else
      return line;
  }
  return "";
}```

@joe-ave
Copy link
Author

joe-ave commented Dec 4, 2023

Hello Olli,

just documenting that priceinfo for tomorrow (2023-12-05) for the nordic countries is still missing at 22:00, probably it will be corrected tomorrow. No info at Entso-e site.

//Jonas

@joe-ave
Copy link
Author

joe-ave commented Oct 5, 2024

Hello Olli,

for info entso-e seem to be having some problem again, both today 2024-10-05 and tomorrow 2024-10-06 prices seem so be missing for all countries. No info at entso-e site.

//Jonas

@joe-ave
Copy link
Author

joe-ave commented Oct 25, 2024

Hello Olli,

Documenting that some priceinfo are missing for tomorrow 2024-10-26 in SE1 to SE4, it looks like 14:00 to 23:00, other countries seem to work.

request sent
headers received
Waiting the document
Debug before get_price_data_entsoe 1729980000, 4848
period_start: 1729720800 record_start: 1729807200 - period_end: 1729980000
ENTSO-E price data missing future prices, end_reached 0, price_rows 38
Price query failed

image

//Jonas

@Olli69
Copy link
Contributor

Olli69 commented Oct 25, 2024

Hi @joe-ave ,
Thanks for reporting! I had a look and could repeat that but did not find a fix yet. It look like a buffer overflow (exists when prices of 3 days are queried) or something else with communication, but strange if that only occurs with SE areas. Anyway this should have automatically fixed before the day with missing prices starts (Arska retries later and then result set from Entso-E contains only prices of 2 days)

EDIT:
Now it should be fixed in the code (not in the binary yet) dev-1.1/src/main.cpp where I added extra wait, if read buffer is empty:

    // If read buffer is empty, wait for a while if new data is coming
    if (!client_https.available()) {
      Serial.println("Waiting new stuff to the buffer");
      delay(1000);
    }

EDIT 2 (Oct 27 2024):
The fix is now included in binary 1.1.0-beta10 available on Arska installation page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants