Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DCF77 Meteo Data 1 full day decode #5

Open
brettoliver opened this issue Jul 11, 2020 · 27 comments
Open

DCF77 Meteo Data 1 full day decode #5

brettoliver opened this issue Jul 11, 2020 · 27 comments

Comments

@brettoliver
Copy link

brettoliver commented Jul 11, 2020

DCF77MeteoDataDecode01..txt

Edit -data is missing from the code I will resend it complete as sson as the new data is ready!

Enclosed 1 full days Meteo Data from 00:00 to 23:59. Maybe you will find this useful.

I have added time to the data at the end of each section using basic serial.print with no leading zeroes added eg 0:3 is 00:03

I am transcribing the data onto a spreadsheet so sections of data can be broken down to see what and when data is sent.

It will take some time to do but can post it if it's of any use.

@tobozo
Copy link
Owner

tobozo commented Jul 13, 2020

Thanks !

My ESP32 can't get a full day from my indoor antenna either, its logs show some holes in the weather data too, and sometimes redundancy between two decodings, which makes me wonder what sort of bug I've created there 🤔

I haven't checked if the DCF77 signal decoding task was still properly timed after the last code changes, it could be a possible regression of a time-shift bug, eventually losing a whole minute of data before being back in sync, or it could just be QRN from the TFT.

Hopefully this will get sorted out after the refactoring.

Anyway, the weathermemory part needs a rewrite now, so I've started storing this data in NVS.

With one full weather data block (=3 bytes) every 3mn, the whole 24h data set can't contain more than 480 entries, some of which may address more than one type of forecast.

I'll go with 480 of those:

struct WeatherForecast {
  int hour;
  int minute;
  uint8_t data[3];
};
WeatherForecast** AllCitiesForecast; // cache all forecasts

and 8 of those for the watched city:

struct WeatherData {

  int hour;
  int minute;

  int dcw;  // Day critical weather
  int ncw;  // Night critical weather
  int xwh;  // Extremeweather
  int anm1; // Weather anomality 1
  int anm2; // Weather anomality 2
  int rpb;  // Rainprobability
  int anm;  // Anomaly
  int tem;  // Temperature
  int twodayforecast;
  int fourdayforecast;
};

I'll probably use 20 slots of another struct for wind directions and strengths, one for each city.

Meanwhile I've been looking for interfaces examples to show DCF77 weather data, and a 160x1218 display obviously can't fit all that.

image

This made me realize using three icons to describe one weather data instance was wrong, so I'll probably, put the wind direction icon in the middle (replacing the atomic icon), remove the rightmost weather icon and reuse the space for windspeed + humidity + temperature.

image

It's a good thing the ButtonC short-push event is currently unassigned, it could end up as a browsing control for the different stored forecasts.

Since the 2-3-4 days forecasts ('DO', 'Fr' and 'SA' on the screen capture) don't have humidity, wind strength and windspeed, the display pattern only requires one icon and one text block, this could be scroll material :-)

@brettoliver
Copy link
Author

I am hoping to get a full days decode by 23:00hrs tonight. I use an Udo Klein Super Filter between my aerial and my ESP32. The filter fills in the gaps if signal is lost. It does not decode the Meteodata though.
I have used a couple of days data to build up a spreadsheet showing what is sent minute by minute. I'll paste a copy here prob Tuesday morning.

I 'll also load your latest code once I have finished recording the serial out. Looks like you have made a number of changes in the last few days.
I like the serial output and have got it working in part on another TFT display using a Arduino as a serial monitor.
Is there a way of making the ESP32 transmit serial at 9600 as this would work better on my serial monitor?

I like your idea of the wind and other data on the display

@tobozo
Copy link
Owner

tobozo commented Jul 13, 2020

Is there a way of making the ESP32 transmit serial at 9600 as this would work better on my serial monitor?

you just need to change this:

M5.begin();

into this:

M5.begin(true, true, false); // LCDEnable, SDEnable, SerialEnable
Serial.begin(9600);

@brettoliver
Copy link
Author

brettoliver commented Jul 14, 2020

Just loaded your latest version. Thanks for all those updates.

I think I have a full days decode now and have included it.
DCF77 Meteo Data Decode13JulyBrett.txt

Have also gone through a days decode minute by minute and put it onto a spreadsheet so the data can be analized.
DCF77 Meteo Data Decode04.xlsx

@brettoliver
Copy link
Author

Thanks for the info on changing the baud rate. I now have a good output to my 2nd TFT serial monitor.
I have tweeked the serial.print outputs to make it fit my display.
terminal
On the TFT display where it says "Rain Showers" I think there is a bit of serial output missing. it should say "Night = 1100 03 Rain Showers"

If you look at my spreadsheet this output is missing on all the Night colums bit there for the day column.
After 21:00 hrs both day and night outputs are complete for the 2 day forecasts.

@tobozo
Copy link
Owner

tobozo commented Jul 15, 2020

Thanks for this data 👍

So if there's only one label missing in the text version, is it safe to assume the weather decoding logic tree is correct ?

I'm rewriting the whole weather display routine where I separate the TFT output from the Serial output (one is more verbose than the other), but there is colateral damage : the binary representation of data chunks is lost.

I'll try to come up later with something to restore that low level debug, but I'll stick with the 32 columns model for the Serial output rendering.

@brettoliver
Copy link
Author

Hi the weather decoding logic looks fine.

I am just waiting for my LOLIN D32 Pro V2.0 to arrive then I can have a play with a 240x320 TFT.

I will just mix and match my serial display output to match your changes as they happen.

@brettoliver
Copy link
Author

Have received my LOLIN D32 Pro V2.0. Compiled and loaded OK to my board.
Do I need to set the pins for my board in your code?

@tobozo
Copy link
Owner

tobozo commented Jul 16, 2020

If you select the corresponding LoLinD32Pro device from the boards menu you don't have any pins to configure provided you're using the 1.4'' TFT model (a ST7735) fitting the special SPI connector, however I'm quite sure it'll also work with an ILI9341 as they share many instructions, and all necessary pins are labelled on the silkscreen.

image

[edit] here's a successful implementation of lolin d-32-pro with an ili9341 with touch spi sharing the tft spi.

image

@brettoliver
Copy link
Author

I have checked and I have the correct board selected.
The code shows the DCF77 pin as 21 and silkscreen on the back of board has this as the SDA pin. I have changed the the DCF77 pin to 13.
I am using a 2.2" ILI9341 TFT. The backlight comes on but nothing is displayed.
The serial out shows
H⸮F�ȭ⸮⸮�� �%�⸮XN�
11Q8⸮⸮)�!⸮Q⸮⸮[E][UI.h:1126] InitUI(): Successfully created spritePtr
[E][UI.h:1133] InitUI(): Successfully created logoSpritePtr

Then nothing else. Do I need to config a 320x240 board in the code?
Any ideas?

@brettoliver
Copy link
Author

Maybe I have the TFT connection wrong?
I had CS on Pin5 SS- moved it to Pin 14 T-CS and now get this

[E][UI.h:1126] InitUI(): Successfully created spritePtr
[E][UI.h:1133] InitUI(): Successfully created logoSpritePtr

Scanning I2C Bus:

 0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
         -- -- -- -- -- -- -- -- -- -- -- -- --

01: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
02: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
03: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
04: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
05: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
06: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
07: -- -- -- -- -- -- -- --
Millistart 12752
Millioffset 0
E (34822) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (34822) task_wdt: - IDLE0 (CPU 0)
E (34822) task_wdt: Tasks currently running:
E (34822) task_wdt: CPU 0: buttonsTask
E (34822) task_wdt: CPU 1: IDLE1
E (34822) task_wdt: Aborting.
abort() was called at PC 0x400e4ccf on core 0

Backtrace: 0x4008c9b4:0x3ffbe170 0x4008cbe5:0x3ffbe190 0x400e4ccf:0x3ffbe1b0 0x40085159:0x3ffbe1d0 0x400dbe12:0x3ffaf630 0x400da404:0x3ffaf650 0x400d7285:0x3ffaf670 0x400d7383:0x3ffaf690 0x4008900d:0x3ffaf6b0

Rebooting...

@tobozo
Copy link
Owner

tobozo commented Jul 17, 2020

Indeed CS should be on 14 as indicated by the "T-CS" label on the back of your LoLin D-32-Pro board.
Does the display flicker during the boot loop or is it always off ?

I can't decode the backtrace for you (it requires the ELF from your compilation) but there's a tool in the Arduino menu where you can paste the whole message and get a hint.

image

Decoding the exception must be done right after compiling or it won't work.
Sometimes the decoded content il also useless or has bad precision but still better than nothing :-)

The exception decoder is available in you "tools" menu / ESP Exception decoder, or you can install it from here.

Another thing you can do is check the pins for the DCF77, also I2C SDA and SCL, and eventually comment out those lines in esp32-DCF77_Analyzer_Clock/Config.h
:

#define USE_RTC
#define USE_BUTTONS

... to see if you still get the same error

[edit] managed to reproduce the same error message after flashing my TTGO-T1 with the wrong board, the timeout was related to the buttons pins not responding, so keeping USE_BUTTONS commented out to see if the display responds is indeed a valid test

@tobozo
Copy link
Owner

tobozo commented Jul 17, 2020

Not exactly a UI enhancement but I'm testing a lighter weather cache system (maintaining one NVS key per city was a bad idea) this time using an array fed with weather data for all cities, and backed up to NVS (max 5Kb) every hour.

image

Note how the last three minutes of every hour never succeed to decode any weather data, same a in your logs.
This seems to affect at least cities 19,39 and 59 and indicates there's a bug somewhere between the signal decoding logic and the tasks management.

The weather display logic is still a work in progress and I need to achieve a visible result before I push the changes, so I'll consider this bug as a tech debt unless you tell me you need the weather data for those specific cities.

@brettoliver
Copy link
Author

Indeed CS should be on 14 as indicated by the "T-CS" label on the back of your LoLin D-32-Pro board.
Does the display flicker during the boot loop or is it always off ?

I can't decode the backtrace for you (it requires the ELF from your compilation) but there's a tool in the Arduino menu where you can paste the whole message and get a hint.

image

Decoding the exception must be done right after compiling or it won't work.
Sometimes the decoded content il also useless or has bad precision but still better than nothing :-)

The exception decoder is available in you "tools" menu / ESP Exception decoder, or you can install it from here.

Another thing you can do is check the pins for the DCF77, also I2C SDA and SCL, and eventually comment out those lines in esp32-DCF77_Analyzer_Clock/Config.h
:

#define USE_RTC
#define USE_BUTTONS

... to see if you still get the same error

[edit] managed to reproduce the same error message after flashing my TTGO-T1 with the wrong board, the timeout was related to the buttons pins not responding, so keeping USE_BUTTONS commented out to see if the display responds is indeed a valid test

Sorry for the late reply have been away....
Have set
DCF77 Pin to 13
commented out USE_BUTTONS
DS1307_SDA to 21
DS1307_SCL to 22
The code now seems run as I am getting serial out
E][UI.h:1126] InitUI(): Successfully created spritePtr
[E][UI.h:1133] InitUI(): Successfully created logoSpritePtr

Scanning I2C Bus:

 0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
         -- -- -- -- -- -- -- -- -- -- -- -- --

01: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
02: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
03: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
04: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
05: -- -- -- -- -- -- -- 57 -- -- -- -- -- -- -- --
06: -- -- -- -- -- -- -- -- 68 -- -- -- -- -- -- --
07: -- -- -- -- -- -- -- --
Meteo Data Decoding Hour 12 Min 50
Meteo Data Decoding Hour 12 Min 51
Meteo Data Decoding Hour 12 Min 52
Meteo Data Decoding Hour 12 Min 53
Meteo Data Decoding Hour 12 Min 54
City 17 Le Havre (France)
Period/Day Day 3
Day Forecast 0100 02Partly cloudy
Clear
Extreme weather 0000 00 None
Rain probability 100 01 15 %
Weather Anomality 0 00 No
Temp 001101 2c 22°C max
Decoder status 10 OK
City not displayed
Hour 12 Min 54

Meteo Data Decoding Hour 12 Min 55
Meteo Data Decoding Hour 12 Min 56

The display however still does not work.
It does flicker a bit then just goes to a grey screen.

@brettoliver
Copy link
Author

re lighter weather cache system
I don't need weather data for those cities so the bug can be left as a tech debt.

@brettoliver
Copy link
Author

Just checking. Is my TFT LOLIND32PRO supported yet in the ESP32-Chimera-Core Library?
Had a quick look in the ESP32-Chimera-Core\boards\d32_pro\variants and it only contains a few pins.
The other boards contain loads more pins.

libraries\ESP32-Chimera-Core\boards\d32_pro\variants
#ifndef Pins_Arduino_h
#define Pins_Arduino_h

#include <stdint.h>
#include <../d32/d32_core.h>

static const uint8_t LED_BUILTIN = 5;
#define BUILTIN_LED LED_BUILTIN // backward compatibility
static const uint8_t _VBAT = 35; // battery voltage

#define TF_CS 4 // TF (Micro SD Card) CS pin
#define TS_CS 12 // Touch Screen CS pin
#define TFT_CS 14 // TFT CS pin
#define TFT_LED 32 // TFT backlight control pin
#define TFT_RST 33 // TFT reset pin
#define TFT_DC 27 // TFT DC pin

#define SS TF_CS

@tobozo
Copy link
Owner

tobozo commented Jul 19, 2020

This variant file is the same as the official one or the other way around if you see how this file was contributed :).

While the LoLinD32Pro is supported by the ESP32-Chimera-Core, the defaut display is the Wemos 1.44'' ST7735 (128x128).
Since it shares the same instruction set as ILI9341 it could work out of the box, but I know the init sequence is slightly different and the screen resolutions don't match so it may mess things up .

The TFT_eSPI setup I was using when I got an ILI9341 (+touch) attached to a LoLinD32Pro was quite short because this configuration is using the default pins from variants.h and doesn't need to be customized much.

I suggest you isolate this problem by having the display running directly with LovyanGFX using the standalone SPI config example first (change line 85 to match your display), then share your custom configuration (the LGFX_Config + panel choice) so I can see why it isn't working out of the box.

[edit]
In LovyanGFX, the pin config is as follows, inheriting from ST7735 which you want to replace by ILI9341 and using some offset you don't need :

  struct Panel_LoLinD32 : public Panel_ST7735S
  {
    Panel_LoLinD32(void) {
      panel_width  = 128;
      panel_height = 128;
      memory_width  = 132;
      memory_height = 132;
      offset_x  = 2;
      offset_y  = 1;
      spi_3wire = true;
      spi_cs    = 14;
      spi_dc    = 27;
      gpio_rst  = 33;
      gpio_bl   = 32;
      pwm_ch_bl = 7;
    }
  };

In order to support ILI9341 instead of ST7735 you probably need to use this config instead :

  struct Panel_LoLinD32Custom : public Panel_ILI9341
  {
    Panel_LoLinD32Custom(void) {
      panel_width  = 320;
      panel_height = 240;
      memory_width  = 320;
      memory_height = 240;
      //offset_x  = 2;
      //offset_y  = 1;
      spi_3wire = true;
      spi_cs    = 14;
      spi_dc    = 27;
      gpio_rst  = 33;
      gpio_bl   = 32;
      pwm_ch_bl = 7;
    }
  };

@brettoliver
Copy link
Author

I am trying the SPI config example. Have set the following
spi_sclk = 18
spi_mosi = 23;
spi_miso = 19;

I am not sure of these settings
spi_host = VSPI_HOST;
dma_channel = 1;
spi_dlen = 8;

@brettoliver
Copy link
Author

Missed your edit will give it a go later today.

@brettoliver
Copy link
Author

Have selected static lgfx::Panel_ILI9341 panel; in LovyanGFX.SPI
and spi_sclk = 18
spi_mosi = 23;
spi_miso = 19;

Have added this ( I presumed not replaced the Panel_ST7735S)

struct Panel_LoLinD32Custom : public Panel_ILI9341
{
Panel_LoLinD32Custom(void) {
panel_width = 320;
panel_height = 240;
memory_width = 320;
memory_height = 240;
//offset_x = 2;
//offset_y = 1;
spi_3wire = true;
spi_cs = 14;
spi_dc = 27;
gpio_rst = 33;
gpio_bl = 32;
pwm_ch_bl = 7;
}
};

Still nothing on the display.

@tobozo
Copy link
Owner

tobozo commented Jul 20, 2020

I just flashed my IL9341/LoLinD32 Pro with the following example sketch and I can see something on the display,
I'm using the LoLin D32 Pro form the boards menu, and the LovyanGFX library is unchanged.

#include "LGFX_TFT_eSPI.h"

// for ESP32
struct LGFX_Config
{
  static constexpr spi_host_device_t spi_host = VSPI_HOST;
  static constexpr int dma_channel = 1;
  static constexpr int spi_sclk = 18;
  static constexpr int spi_mosi = 23;
  static constexpr int spi_miso = 19;
  static constexpr int spi_dlen = 8;
};


static lgfx::LGFX_SPI<LGFX_Config> lcd;
static lgfx::Panel_ILI9341 panel;


void setup(void) {

  panel.freq_write = 20000000;
  panel.freq_fill  = 27000000;
  panel.freq_read  = 16000000;
  panel.spi_mode = 0;
  panel.spi_mode_read = 0;
  panel.len_dummy_read_pixel = 8;
  panel.spi_read = true;
  panel.spi_3wire = false;
  panel.spi_cs = 14;
  panel.spi_dc = 27;
  panel.gpio_rst = 33;
  panel.gpio_bl  = 32;
  panel.pwm_ch_bl = 7;
  panel.backlight_level = true;
  panel.invert = false;
  panel.rgb_order = false;
  panel.memory_width  = 320;
  panel.memory_height = 240;
  panel.panel_width  = 320;
  panel.panel_height = 240;
  panel.offset_x = 0;
  panel.offset_y = 0;
  panel.rotation = 0;
  panel.offset_rotation = 0;

  lcd.setPanel(&panel);
  lcd.init();

  lcd.drawRect(0,0,lcd.width(),lcd.height(),0xFFFF);
  lcd.setTextSize(2);
}

uint32_t count = ~0;

void loop(void) {
  lcd.startWrite();
  lcd.setRotation(++count & 7);

  lcd.setTextColor(random(65536));
  lcd.drawNumber(lcd.getRotation(), 16, 0);

  lcd.setTextColor(0xFF0000U);
  lcd.drawString("R", 30, 16);
  lcd.setTextColor(0x00FF00U);
  lcd.drawString("G", 40, 16);
  lcd.setTextColor(0x0000FFU);
  lcd.drawString("B", 50, 16);

  lcd.drawRect(30,30,lcd.width()-60,lcd.height()-60,random(65536));
  lcd.endWrite();
}

@brettoliver
Copy link
Author

have loaded the program no change. I then changed the TFT to a new one ( I have several and have tried a few before today) and got this image
test

I then loaded LovyanGFX SPI and got the same image.

I then tried your DCF77 analyzer and could see the menu flasing in the corner of the display then it loaded the main analyzer pic then went off again very small then the screen went grey again.
analyzer

@brettoliver
Copy link
Author

With the analyzer loaded the following is output from the serial port
Scanning I2C Bus:

 0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
         -- -- -- -- -- -- -- -- -- -- -- -- --

01: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
02: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
03: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
04: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
05: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
06: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
07: -- -- -- -- -- -- -- --
Millistart 12543
Millioffset 0
E (34608) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (34608) task_wdt: - IDLE0 (CPU 0)
E (34608) task_wdt: Tasks currently running:
E (34608) task_wdt: CPU 0: timerTasks
E (34608) task_wdt: CPU 1: IDLE1
E (34608) task_wdt: Aborting.
abort() was called at PC 0x400e4cb7 on core 0

Backtrace: 0x4008c9b4:0x3ffbe170 0x4008cbe5:0x3ffbe190 0x400e4cb7:0x3ffbe1b0 0x40085159:0x3ffbe1d0 0x40102c9c:0x3ffce470 0x400dbc2a:0x3ffce490 0x40104022:0x3ffce4d0 0x400d4517:0x3ffce510 0x400d70c0:0x3ffce580 0x4008900d:0x3ffce5b0

Rebooting...

@brettoliver
Copy link
Author

I can't see the ESP exception decoder under the tools menu.

@tobozo
Copy link
Owner

tobozo commented Jul 20, 2020

yay some progress ^^

you can find this line somewhere in UI.h and comment out the #if / #endif around it :

#include "UI_320x240.h"

The exception decoder can be downloader here : https://github.com/me-no-dev/EspExceptionDecoder

I'll see how I can integrate LGFX configuration to this project, but I'm still refactoring the weather decoding so it'll take some time.

@brettoliver
Copy link
Author

OK thanks will have a look in the morning and let you know. Thanks for the help.

@brettoliver
Copy link
Author

Have set #include "UI_320x240.h" as above but display remains the same.

I will leave you to complete refactoring the weather decoding before before continuing with the LGFX problem.
I'll work on my schematics on the D32pro/arduino serial port display.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants