Observations on SPI/EPG #336
Comments
|
Hello,
The thing I noticed with converting EPG output to xml was that this
conversion - using the code that was copued from the sources by Jiliam
Cablr - the software - i.e. the whole Qt-DAB program - crashed.
I did not try to debug that code, since I am processing the binary data,
there was no need, it was added on a request of a user.
FYI Here in the netherlands, we do not have an EPG service, so testing is
done on recordings. Some people were kind enough to send some recordings.
So, I am not able to process files from the local neighborhood.
Is\f your question is obtaining am ".ehb" file than it would be helpful to
send a recording from which I can dervice such a file
Just an issue about netetiquette
Normally I do not react on emails from people who do not show their name
Best
jan
Op zo 3 nov 2024 om 17:07 schreef Confused-52 ***@***.***>:
… I discovered Qt-DAB a couple of days ago in my quest for something to help
me recreate EPG data for Pure EVOKE 3, which sometimes goes missing on the
UK’s BBC National Mux.
1. You say that errors have been reported in the SPI xml produced.
Firstly it needed information from Welle.io discussions to tell me what
changes I needed to get spi xml to be produced. I generated a set of xml
files and compared them to the DAT files held by the Pure radio,
The errors I found were all related to the transposition of non Ascii
compatible UTF-8 characters. It seems likely that these problems originate
in epg-decoder.cpp where string manipulation uses conversion to and from
UTF8. I fear this is an over complication which causes errors.
Clause 4.5.1 of TS102371 the binary file format states that strings
are to use: “ISO/IEC 10646 [4] with UTF-8 encoding. The ISO/IEC 10646 [4]
characters
0xE000 to 0xF8FF shall not be included within any binary encoded
strings.”
TS102818 the xml file format says: “The ISO/IEC 10646 [19] character
set using UTF-8 character encoding must be used in all EPG XML documents
where
applicable.
NOTE: The ISO/IEC 10646 [19] character set contains all characters of the
DAB character sets (three EBU
Latin-based sets, ISO 8859-2 [20] and ISO/IEC 10646 [19] using UTF-8).”
As in all ETSI documents the NOTE is informative and just tells us that
all the relevant characters are included for European use.
So the bottom line is that the character string in the downloaded .EHB
file, which is tagged as shortDescription inside the mediaDescription tag,
should be copied unchanged.
Having the .EHB file in the Pure SDcard I copied the origin two byte UTF8
character causing trouble in the SPI files into the SPI files and the XML
editor and edge both stopped finding faults.
2) I would find it helpful if I could obtain the downloaded binary .EHB
files to examine as well as the xml files, clearly in a different directory
would be helpful. The was a request for the service details and date on an
earlier request and there is sufficient information in the .EHB to follow
the informative Annex C clause 5.1.1 of TS102818 i.e YYYYMMDD___PI.xml
from the serviceID element. Pure fail on this but provide a directory with
conversions from the Downloaded names to a shortened form of Annex C
naming, omitting the Country code.
I am running a windows10 precompiled version so cannot provided changes to
the code. My only Linux machine is RPI 4 so the cross-compile route fails
to appeal. I hope this interest enables yo to carry on working on the epg
side of the project.
—
Reply to this email directly, view it on GitHub
<#336>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACCPHQEHZCEZKH7CA762TXDZ6ZC33AVCNFSM6AAAAABRC3QJNOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGYZTCMZQGI2TMMQ>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
--
Jan van Katwijk
|
|
Jan, Thanks for your reply. I could see that you had not used Julian Cables code from the syntax file and comments in epg-decoder.cpp in folder epg-2 of the source. I have attached a zip containing the derived xml from a run of Qt-DAB and the equivalent EHB file which is the binary form you extract data from. This .EHB file is taken from the Directory on the Pure Evoke 3 and was saved with the same Transport id value which was associated to the xml file both within a short timeframe so we know they are equivalent. The files can be examined and the xml fault in all the files which a browser reported as faulty were the same logical error. That is there was an instance of a two byte UTF-8 character transcoded into a single byte character which was invalid UTF-8. I hope that this allows you to find where the corruption is occurring. (Dara Ó Briain is the man's real name) It would be nice to get the EHB files but you are already isolating them as the binary data you take from the MOT carousel and recognise as schedules. The pure data may be just a subset but it would be clear if you captured the whole MOT object and saved it with Transport IDBinary as a name. I am using Windows 10 on a surface 3 and it does not seem to have sufficient power to capture a sufficiently long recording to capture a full MOT carousel. The BBC National dab service sends 77 EHB files plus the directory as well as a number of png files. The problem I am trying to solve is that occasionally the BBC fail to send EPG data at all and my wife uses it to record programmes on that Evoke radio. There are other bits of software around in java from Global Radio to produce .EHB files from xml spi. Now the data that goes into the schedule is available from other BBC sources which are more reliable, so I am wondering if I can reverse engineer that reliable data into the a directory and EHB files that will work for just one day on a single service. A further observation on what I have seen is that Qt-DAB ha never produced more than 72 spi files for me and I wonder if it would be better recto recreate the MOT directory rather a rolling buffer of 16 files. I can indeed see that you would be better able to do that with a recording containing a full set of data. If I can stabilise the situation which king of data file would be best for you? Best Regards, Phil [Note.zip](https://github.com/user-attachments/files/17650540/Note.zip |
|
Thanks for the info.
EPG gives me a little headache since as said, here there is no dutch
channel with an ensemble that contains EPG type data.
Up to a few years ago the channel of the dutch NPO had an EPG service, but
that is gone.
I do have a couple of recording (usually a few minutes) with and EPG
service, but there I can decode some fragments,
So, all working on the code right now is "paper work", no possibilities of
testing.
Best
jan
Op wo 6 nov 2024 om 19:16 schreef Confused-52 ***@***.***>:
… Jan,
Thanks for your reply.
I could see that you had not used Julian Cables code from the syntax file
and comments in epg-decoder.cpp in folder epg-2 of the source. I have
attached a zip containing the derived xml from a run of Qt-DAB and the
equivalent EHB file which is the binary form you extract data from. This
.EHB file is taken from the Directory on the Pure Evoke 3 and was saved
with the same Transport id value which was associated to the xml file both
within a short timeframe so we know they are equivalent. The files can be
examined and the xml fault in all the files which a browser reported as
faulty were the same logical error. That is there was an instance of a two
byte UTF-8 character transcoded into a single byte character which was
invalid UTF-8. I hope that this allows you to find where the corruption is
occurring. (Dara Ó Briain is the man's real name)
It would be nice to get the EHB files but you are already isolating them
as the binary data you take from the MOT carousel and recognise as
schedules. The pure data may be just a subset but it would be clear if you
captured the whole MOT object and saved it with Transport IDBinary as a
name.
I am using Windows 10 on a surface 3 and it does not seem to have
sufficient power to capture a sufficiently long recording to capture a full
MOT carousel. The BBC National dab service sends 77 EHB files plus the
directory as well as a number of png files.
The problem I am trying to solve is that occasionally the BBC fail to send
EPG data at all and my wife uses it to record programmes on that Evoke
radio. There are other bits of software around in java from Global Radio to
produce .EHB files from xml spi. Now the data that goes into the schedule
is available from other BBC sources which are more reliable, so I am
wondering if I can reverse engineer that reliable data into the a directory
and EHB files that will work for just one day on a single service.
A further observation on what I have seen is that Qt-DAB ha never produced
more than 72 spi files for me and I wonder if it would be better recto
recreate the MOT directory rather a rolling buffer of 16 files. I can
indeed see that you would be better able to do that with a recording
containing a full set of data. If I can stabilise the situation which king
of data file would be best for you?
Best Regards,
Phil
[Note.zip](https://github.com/user-attachments/files/17650540/Note.zip
—
Reply to this email directly, view it on GitHub
<#336 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACCPHQHR7XOCLTN6NI6MUY3Z7JMHFAVCNFSM6AAAAABRC3QJNOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRQGQ3DOMZRHE>
.
You are receiving this because you commented.Message ID:
***@***.***>
--
Jan van Katwijk
|
I discovered Qt-DAB a couple of days ago in my quest for something to help me recreate EPG data for Pure EVOKE 3, which sometimes goes missing on the UK’s BBC National Mux.
The errors I found were all related to the transposition of non Ascii compatible UTF-8 characters. It seems likely that these problems originate in epg-decoder.cpp where string manipulation uses conversion to and from UTF8. I fear this is an over complication which causes errors.
Clause 4.5.1 of TS102371 the binary file format states that strings are to use: “ISO/IEC 10646 [4] with UTF-8 encoding. The ISO/IEC 10646 [4] characters
0xE000 to 0xF8FF shall not be included within any binary encoded strings.”
TS102818 the xml file format says: “The ISO/IEC 10646 [19] character set using UTF-8 character encoding must be used in all EPG XML documents where
applicable.
NOTE: The ISO/IEC 10646 [19] character set contains all characters of the DAB character sets (three EBU
Latin-based sets, ISO 8859-2 [20] and ISO/IEC 10646 [19] using UTF-8).”
As in all ETSI documents the NOTE is informative and just tells us that all the relevant characters are included for European use.
So the bottom line is that the character string in the downloaded .EHB file, which is tagged as shortDescription inside the mediaDescription tag, should be copied unchanged.
Having the .EHB file in the Pure SDcard I copied the origin two byte UTF8 character causing trouble in the SPI files into the SPI files and the XML editor and edge both stopped finding faults.
2) I would find it helpful if I could obtain the downloaded binary .EHB files to examine as well as the xml files, clearly in a different directory would be helpful. The was a request for the service details and date on an earlier request and there is sufficient information in the .EHB to follow the informative Annex C clause 5.1.1 of TS102818 i.e YYYYMMDD___PI.xml from the serviceID element. Pure fail on this but provide a directory with conversions from the Downloaded names to a shortened form of Annex C naming, omitting the Country code.
I am running a windows10 precompiled version so cannot provided changes to the code. My only Linux machine is RPI 4 so the cross-compile route fails to appeal. I hope this interest enables yo to carry on working on the epg side of the project.
The text was updated successfully, but these errors were encountered: