Migrate files from CVS to Girr

[bengtmartensson]

The current CSV based file format of IRDB has several shortcomings:

First, using the protocol and its device & subdevice as the database key is problematic:

1. The (inexperienced) user looking for codes for their device is sort-of required to know the answer to look for it.

2. Several devices (Sony TV/Projectors, Yamaha AV receivers, Philips TVs, Denon Receivers come to my mind) support commands with different device/subdevice numbers, and even different protocols. Such a device cannot be contained in one file, but requires several. Worse yet, how is the owner of, say, a Sony TV (my AF9 has commands using the sony12, sony15 and sony20 protocols) know how to combine files to control their device? Again, the user is (sort-of) required to know at least a partial answer already.

The CSV format is a very simplistic file format. As used in IRDB, there have the following problems:

1. Not all protocols use device ("D"), subdevice ("S") and function ("F") as their parameters. Using -1 for non-existing is ... ugly. Many protocols use more parameters, and they cannot really be covered.

2. The files as such cannot contain meta data. Instead, in IRDB, some of this information is contain in the path of the file. This is very limited, and not very data base friendly.

Instead, I propose to use the Girr format, or at Github. This is a very versatile XML-based format that can (optionally) contain all sort of different representations like raw and Pronto Hex. The format as such (including documentation and its machine readable grammar (W3C Schema) is public domain. There is an accompanying Java library, which is GPL3.

Girr is the native format of IrScrutinizer. I am happy to report that is is also supported for import and export in the current version of the program RemoteMaster used in the JP1 community. (Quite old manual). It is thus quite easy to program a "JP1-supported" remote from a Girr file, or to generate a Girr file from a "JP1-upgrade".

I have a small collection of Girr files. This can serve as an example of how an IRBD-like server can be organized. (Actually, it would be preferable to have am intermediate directory, denoting device class ("TV", "DVD" etc).)

Using the aforementioned library, it would be a relatively easy task (1 day?) to write a program that converts the existing CVS files to corresponding Girr files. In the first translation, can keep the
present key (protocol, device, subdevice) as the key, but in subsequent (manual) fixes, it would be desirable to fix.

(The other direction transformation is easy; here is an XSLT-script that generates a IRDB-style CSV fromm a Girr.)

[probonopd]

So I think we need a sane architecture for the input (how do people submit new codes to irdb) and output (which formats we generate).

For the input, I liked CSV because it is easy to understand and people should be able to submit new codes relatively easily. I now understand that maybe it is not the best input format. What should we use instead?

For the output, we could have a GitHub Action that would run a conversion from the input format to output format(s), e.g., Girr, maybe a SQLite database that IrScrutinizer could use, etc.

Worse yet, how is the owner of, say, a Sony TV (my AF9 has commands using the sony12, sony15 and sony20 protocols) know how to combine files to control their device?

Do you know how commercial databases (e.g., the ones used by One For All) handle these "multiple protocol" remotes?

[bengtmartensson]

For the input, I liked CSV because it is easy to understand and people should be able to submit new codes relatively easily. I now understand that maybe it is not the best input format. What should we use instead?

First of all, it is not the most important question, at least not now. There has not really been that many submissions in the past... The coolest alternative would be to be able to submit directly from IrScrutinizer, a button called "Submit parametric Remote to IRDB", but it would of course require the server to have an appropriate API call. It is possible to tell people to generate Girr from, say, CSV using IrScrutinizer, or a supplied command line program. All submissions methods (except for manual hand-holding) have to adhere to some formal procedures...

For the output, we could have a GitHub Action that would run a conversion from the input format to output format(s), e.g., Girr, maybe a SQLite database that IrScrutinizer could use, etc.

Well, right now nothing is working, also not before the resent demise of irdb.tk... It all depends on the server. (BTW, here is a project that may be useful, in parts.)

[lbschenkel]

I would like to reiterate that it's a good thing to lower the barrier of submission as much as possible, so CSV even with all its drawbacks at least allows somebody to submit new codes that they happen to know without having to download/use some extra software. (Don't get me wrong, IrScrutinizer is amazing, I use it myself.)

That said, maybe one way to work around that is to create a GitHub issue template so users that cannot or will not use IrScrutinizer can submit new codes by creating an issue, filling up the form (the form could allow entering the parametric data and/or Pronto hex) and a maintainer has to put those in.

Another thing that CSV has for it is that in the absence of a generated web site, you can easily see all the parametric codes directly in GitHub. If those were Girr files then doing XML parsing by head is not as friendly (even though it's possible).

Maybe some of those concerns could be alleviated by creating a YAML flavour of Girr that contains the same data as the XML representation but it's more human readable and more diffable. It would then retain all the power of Girr but also keeping the property of being easily readable/modifiable without the need of extra software.

[lbschenkel]

Girr is the native format of IrScrutinizer. I am happy to report that is is also supported for import and export in the current version of the program RemoteMaster used in the JP1 community.

I use both myself but unfortunately RemoteMaster has a bug where it discards all the comments when reading Girr files (exporting a device upgrade works fine, the comments are preserved). I am looking at the source and trying to fix that myself in order to submit a fix upstream.

[probonopd]

First of all, it is not the most important question, at least not now. There has not really been that many submissions in the past... The coolest alternative would be to be able to submit directly from IrScrutinizer, a button called "Submit parametric Remote to IRDB", but it would of course require the server to have an appropriate API call.

Agree!

GitHub has an API that should make it possible to open a pull request maybe? But that'd require potential contributors to have a GitHub account.

Or maybe someone could operate a small web service that would receive newly submitted codes and make pull requests in a server2server way. Just brainstorming here.

[bengtmartensson]

GitHub has an API that should make it possible to open a pull request maybe? But that'd require potential contributors to have a GitHub account.

Won't work that easily. First, the way git work requires you to (assuming you have an account) fork the original repository, commit to your fork, then creating a PR. Then you have to generate a Github token... Does not sound like a really great idea, at least not on the submitter's host.